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ABSTRACT 



The progress made by adults in dedicated mainstream basic 
skills provision in England and Wales and factors associated with students ' 
progress were examined. Of the 2,135 students from 71 colleges of further 
education and local education authorities who took the reading pretest, 1,224 
(57%) took the reading posttest. Writing scripts were received from 1,724 
students at pretest and 937 (54%) at posttest. Background data were collected 

on the students, and 177 adult literacy tutors completed a questionnaire. 
Students achieved small but statistically significant improvements in reading 
and very small but statistically significant improvements in writing. 
Students’ writing improved only in terms of length of script and quality of 
handwriting- -not in terms of reduction of errors or increase in complexity. 
Students' writing skills were generally much poorer than their reading 
skills. No factors associated with differential progress in writing were 
found. The following factors appeared linked to differential progress in 
reading: regular attendance, all tutors in an area having qualified teacher 
status, and tutors having help in the classroom. Nine recommendations were 
presented. (Sixty-one tables/figures and 22 references are included. The 
following items are appended: descriptions of the study methodology and 
method used to devise the reading tests, the student profile instrument, and 
the tutor questionnaire.) (MN) 
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Foreword 



Although there have been adult literacy programmes for many years, we know 
surprisingly little about whether adults do improve these crucial literacy skills in 
such programmes. There was a national study when adult literacy provision 
expanded in the mid-1970s but there has been little since. Furthermore, there have 
been few reliable studies of learner progress in other countries, even in the United 
States. 

The NFER research in the mid-1970s suggested that about 25% of learners made 
significant progress, 50% made only very modest progress and 25% ‘stood still’. (So 
many adults had ‘dropped out’ by the time the research was completed that there’s 
some doubt about the reliability of the results.) It would be encouraging to report 
that this recent NFER study showed many more adults making substantial progress 
and far fewer ‘standing still’. It would be encouraging but it wouldn’t be true. 

This research suggests that most adults make modest yet worthwhile progress in 
reading. On the other hand most don’t appear to make any measurable progress in 
writing, particularly in terms of improved spelling and punctuation, yet a well 
written and correctly spelled application is often the difference between getting a job 
and remaining unemployed. 

Of course, there are plenty of reasons why progress has been so modest. Most adults 
get only a few hours of teaching a week. Provision has been poorly funded and has 
usually been seen as a ‘Cinderella’ service of low priority. Just as importantly the 
adults in literacy courses failed to master these essential skills in school and require 
skilled teaching if they’re to master them later in life. 

However, it’s no good bemoaning the lack of priority given in the past or the very 
limited access to training most teachers of literacy to adults have had. What we 
need to do is make sure that the new government adult basic skills strategy is so 
effective that any adult joining a literacy course is guaranteed to make good 
progress. Such good progress that they will not need similar help in future. If we can 
meet this challenge, a future research report on progress will be a real cause for 
celebration. 

Alan Wells 
Director, Basic Skills Agency 




Executive Summary 



PRINCIPAL FINDING 

• Adults in dedicated mainstream basic skills provision in England and Wales made 
undramatic but worthwhile progress in both reading and writing. 

• At pre-test, just under half the sample had reading scores below a functional 
level. 

• Many students seemed to have distinctly poorer writing than reading skills. 

• No factors associated with differential progress in reading were found, and only 
three factors associated with differential progress in reading: regular attendance, 
all tutors in an area having qualified teacher status, and tutors having help in the 
classroom. 



Method 

• The Basic Skills Agency commissioned the National Foundation for Educational 
Research to investigate the progress in literacy made by adults in dedicated 
mainstream basic skills provision in England and Wales, and factors associated 
with that progress. 

• The study began in early 1998, and lasted two years. 

• Thus it began before, but was completed after, the publication of the Moser 
Report. 

• A set of reading tests was compiled from items used in previous studies. The items 
were chosen, and the tests designed, to provide three forms of national data: 
norm-referenced results against a national distribution, criterion-referenced 
results against a set of basic skills standards, and comparisons with school-level 
performance. 

• Two main reading test forms were devised. Half the sample (randomly assigned) 
took each form at pre-test, and the other form at post-test. A complex and 
innovative statistical process was used to equate the test forms. The tests made 
literacy demands which could be considered largely functional. 

• One-sentence writing prompts were also used at pre- and post-test. 



2 Progress in Adult Literacy 



• A total of 2,135 students (representing provision in 71 Colleges of Further 
Education and Local Education Authorities) took the reading pre-test, and 1,224 
also took the reading post-test (57 per cent retention). Writing scripts were 
received from 1,724 students at pre-test, and 937 at post-test (54 per cent 
retention). Detailed comparisons showed that those who returned at post-test 
were fully representative of the whole pre-test sample test, in terms of the 
characteristics on which information was available. 

• Most of the students who took part in the study were in 'normal’, ongoing basic 
skills provision. However, a small number of experimental and highly intensive 
courses were mounted in March 1999, and 206 students who attended those 
courses were studied both in their own right and as part of the full sample. 

• Background data were gathered on students, and 177 adult literacy tutors 
responded to a questionnaire. 

• Qualitative information was derived from interviews with seven basic skills co- 
ordinators and from fieldworkers’ serendipitous impressions - there was no 
opportunity to carry out systematic observations of teaching. 

Previous studies 

• None of the previous national studies of progress in adult literacy in Britain or the 
USA provided sufficiently reliable data. 

• It therefore appears that the study reported in this volume is the first in the 
English-speaking world to provide reliable evidence of progress in adult literacy 
based on an adequately representative national sample. 



Adult literacy teachers and teaching in England and Wales 

• Information from a questionnaire returned by 177 adult literacy tutors provided 
a sketch of adult literacy teachers and teaching. 

• Adult basic skills is a largely part-time and very largely female profession. Tutors 
are predominantly qualified in Arts subjects and mostly teach small groups a few 
hours per week. 

• Adult literacy tutors seemed to keep largely up to date in terms of attending 
courses and reading books relevant to students’ needs, but the writers on literacy 
who were most familiar to them had produced their most influential work some 
time ago. 

• Overall, professional development and management support for tutors appeared 
patchy. 
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• It seemed that appropriate use was being made of classroom assistants, where 
they were available. No explanation was available of why nearly two fifths of 
tutors had no assistance. 

• The overall impression of teaching was one of flexibility and individual initiative. 
This could be seen either as professionalism or as 'winging it' - an approach that 
may be forced on tutors by the highly part-time nature of their work. 

The sample 

The full sample of students in this study: 

• contained significantly more women than men; 

• represented a range of ages and occupational statuses; 

• contained minorities of people for whom English was an additional language, and 
of ethnicities other than white; 

• were mostly poorly qualified; 

• had mostly attended basic skills provision before, and a minority were attending 
more than one form of provision in 1998/99; 

• mostly received more than 30 hours of tuition between pre- and post-test; 

• had reading levels quite similar, on average, to a national sample of 9-year-olds; 

• were judged by their tutors to range from very weak to about the national 
average. 



The main findings 

• The students achieved a small but statistically significant improvement in 
reading. 

• The students achieved very small but statistically significant improvements in 
writing, but only in terms of length of script and quality of handwriting. No 
significant improvements were found in terms of reduction of errors or increase in 
complexity. 

• These findings were true both for the full sample and (except for handwriting) for 
a subsample of 206 students who took part in highly intensive courses in March 
1999. 

• In reading, over half the sample were already reading at a functional level at pre- 
test, and about one student in 10 was already close to the national norm. 
However, about three in ten had very low pre-test scores. 



4 Progress in Adult Literacy 



• Even so, these findings suggest that an even smaller proportion of adults in 
greatest need of help with literacy are being reached than previously thought, 

• Many students seemed to have developed coping strategies for reading non- 
continuous 'official’-type texts, but could cope less well with continuous texts 
such as newspaper articles. 

• The students in the study may be unrepresentative of adults in the lower half of the 
national distribution of literacy attainment in this disjunction of reading skills. 

• Many students seemed to have distinctly poorer writing than reading skills. 

• Several of these findings lead to the conclusion that many students' literacy 
attainments are uneven, between reading and writing, between types of reading 
material, and from one occasion to another. 

• Some students seemed to attend as much to maintain their skills as to improve 
them. This may be particularly true of those who have learnt to cope better with 
non-continuous than with continuous texts. 



Factors associated with progress 

• The students' gains were statistically related to their background characteristics, 
to the amount of tuition they had received between pre- and post-test, and to 
important variables from the tutor questionnaire. 

• In writing, no clear pattern of factors associated with progress emerged. 

• The analysis against background characteristics showed that virtually all 
subgroups of the full sample made progress in reading, but very few factors 
associated with differential progress were discovered. 

• Students who attended 51-60 hours of literacy tuition between pre- and post-test 
made the largest gain of any subgroup. Since the maximum number of teaching 
weeks between the tests was about 20, these were students who had attended 
very regularly. 

• Since the students on intensive courses by definition attended regularly, the 
finding on the benefit of regular attendance suggests that intensive courses could 
be particularly effective. 

• In the analyses against tutor questionnaire variables, most factors produced non- 
significant results, and only two factors were discovered to be significantly 
associated with progress. 

• Where all the tutors in a particular provider’s area had qualified teacher status, 
students made significantly more progress than in other areas. 
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• Where tutors had help in the classroom (volunteers or paid assistants), students 
made significant progress, but where tutors had no assistance students on 
average made no progress. 

• Interviews with co-ordinators in seven areas where students made statistically 
significant progress revealed little that was unexpected, except perhaps a 
tendency towards more detailed attention to staff induction and development. 

• A number of findings suggested a need for greater professionalisation of the adult 
basic skills teaching force. 

Recommendations 

• All new students should be encouraged to attend very regularly, perhaps on an 
intensive course/provision. Intensive provision should perhaps also be offered to 
existing students who want to try getting a quick boost to their skills. 

• Many students will have the further goal of not losing the skills they have gained, 
and these students will need both maintenance of their skills and support for 
their literacy dealings with the world. They will continue to need non-intensive 
provision. 

• In-service professional development for tutors should embed attention to subject 
knowledge of the practicalities of literacy (teaching spelling, phonemic awareness 
and a range of cueing systems, comprehension, etc.) in a broader context of up- 
to-date theory, and aspects of pedagogy specific to literacy should receive more 
emphasis than general principles of teaching. Access to in-service provision 
should be eased for the large majority of the profession who are part-time. 

• Improved professionalisation of adult literacy tutors should be designed to lead to 
more direct teaching of the specifics of literacy. 

• There is a need for a planned series of micro-studies, of which four are suggested 
below. 

• Methods of making classroom assistance more widely available should be 
examined, and the best ways to deploy tutors’ classroom assistants should be 
investigated. 

• Instruments for assessing students' progress need to be able to detect very small 
steps of progress at the lowest levels. 

• The conditions in which students' learning is best consolidated and maintained 
need to be much better understood. 

• Reasons for the apparent discrepancy between reading and writing attainment 
should be investigated, along with its implications for teaching. 



Chapter 1 



Context, aims and outline of the 
study 



1.1 Context 

The context in which this report is published differs significantly from that in which 
the study was launched - because of A Fresh Start: Improving Literacy and Numeracy 
(Moser Report, 1999) and its inauguration of a new era in adult basic skills in 
England, and Improving Standards of Literacy and Numeracy in Wales: a National 
Strategy (Basic Skills Agency, 2000) with its proposals for Wales. The study began as 
a data-gathering exercise within a basic skills context of ‘business as usual’, though 
the ‘business’ was steadily expanding. This report was published after the two reports 
put basic skills in the educational and political spotlight, and set ambitious targets for 
the expansion of provision. In both countries new structures were to be established. 
In England, a Learning and Skills Council (LSC) was to come into existence in April 
2001, together with 47 local LSCs, and subsume many of the functions of the 
Further Education Funding Council and the Training and Enterprise Councils. For 
Wales, Community Consortia for Education and Training were proposed. Each 
country was to have a national curriculum and a national test for basic skills and 
enhanced training for basic skills teachers. The results reported here may therefore 
contribute more than had been envisaged to the development of policy and practice. 

Through the years of ‘business as usual’, the Basic Skills Agency (BSA) consistently 
encouraged greater take-up of adult literacy provision in England and Wales: for 
instance, between 1995/96 and 1996/97 alone, the number of people enrolled in 
provision for adult literacy (alone, or combined with numeracy) and English as an 
Additional Language (EAL) rose from 328,801 to 362,449, an increase of 10.2 per 
cent (BSA, 1999, Table 5, p.3). The Agency also encouraged the spread of good 
practice in basic skills tuition, in the assessment and accreditation of students, and 
in the accreditation of programmes and staff development. It published materials 
and frameworks for student assessment, and commissioned research into student 
retention, dropout and progression, into the effectiveness of family literacy and 
workplace programmes, and into the impact of basic skills support in Colleges of 
Further Education on student retention and achievement. 



But a significant gap in the BSAs knowledge remained: there was little information 
on the effectiveness of basic skills provision, that is, on the amount of progress which 
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adults make in such provision. The previous information was both limited and dated 
(see chapter 2). In early 1998, the Agency therefore commissioned research into 
this area from the National Foundation for Educational Research (NFER). The study 
began in early 1998 and lasted two years. 

1.2 Aims 

The aims of the study were to investigate: 

• the progress made in literacy by adults in dedicated basic skills provision offered 
by Colleges of Further Education (FE Colleges) and Local Education Authorities 
(LEAs) in England and Wales, and 

• factors associated with improvement in literacy by adults in such provision. 

‘Dedicated’ meant provision in which improvement in basic skills was the students’ 
primary objective and not, for instance, basic skills support for students whose main 
programme was some other course of study. The term ‘provision’ is used here 
instead of ‘programmes’ or ‘courses’ in order to include drop-in provision and one- 
to-one college tuition in addition to timetabled courses for groups of students. The 
study was intended to cover dedicated adult literacy provision (with or without 
numeracy), whether delivered in timetabled group provision, open learning, drop- 
in, or one-to-one tuition in an FE College or LEA centre with a tutor (with or 
without support from volunteers or paid assistants). 

There were also various exclusions from the study’s remit. It was not designed to 
cover numeracy, basic skills support as an adjunct to other courses, family literacy 
programmes, employment-related courses, literacy schemes in prisons, one-to-one 
home tuition, literacy in Welsh, or distance learning. It was also not designed to cover 
provision that was solely for people with EAL or who had learning difficulties, though 
where such students were attending more general provision they were to be included. 

The study was to be designed to provide answers not only to the question ‘How 
much progress do adult literacy students make?’, but also to the question ‘How good 
is their progress in national terms?’ ‘In national terms’ meant that the study should 
provide: 

• norm-referenced statistics to answer the question ‘How good was the progress 
made by these students in relation to national adult norms for literacy?’ 

• some comparison with the performance of school pupils 

• criterion-referenced statistics to answer the question ‘What proportion of the 
students tested moved up at least one level on a relevant set of literacy Standards?’ 

This was to be a progress study, designed to estimate any change in attainment that 
was occurring in ‘normal’, ongoing, typical adult literacy provision. It was not an 
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intervention or impact study, in the sense that there was no attempt to introduce 
any innovations or special initiatives into the situation, still less to evaluate any such 
innovations. 

There was a partial exception to this, however. In early 1999, the BSA received 
approval to fund a small number of intensive basic skills courses. These were to 
begin in late February, provide 60 hours of tuition, and be completed by the end of 
March. Nine providers in England agreed to mount three courses each, with each 
course group numbering 10 students. These courses used a pilot version of the new 
curriculum for adult basic skills which was due to be published in 2000, and 
incorporated more direct teaching of literacy skills than was evident in conventional 
provision (see Chapter 3). 

The Basic Skills Agency commissioned NFER to measure the progress of students in 
these courses too. (A parallel qualitative evaluation was also commissioned - see 
Basic Skills Agency, 2000.) It was further agreed between the BSA and NFER that 
this group of students would not only be studied in their own right, but be counted 
towards the existing study. In a sense, the intensive courses constituted a small 
‘experimental’ condition within what was otherwise purely a progress study; 
although there had been intensive courses before, this was the first attempt to 
mount a number of such courses simultaneously and study them systematically. 

The intensive courses were studied in exactly the same way as the forms of provision 
which contributed to the ongoing study, and this report covers progress in both 
'normal’ provision and the intensive courses. Where necessary the two aspects of 
the study are distinguished by referring to the ‘full sample’ and ‘the intensive 
students/sample’. The students in the intensive sample numbered 206. 

1 . 3 Outline of how the study was carried out 

(A full description of how the study was carried out is given in the Appendices.) 

About 30 adult basic skills co-ordinators attended consultation meetings at the 
Agency to help shape the project. Their input was crucial to the design of the 
reading tests, and to smoothing the fieldworkers’ paths. 

A set of reading tests was compiled from items used in previous studies: 

• the International Adult Literacy Survey (IALS) of 1996 (Carey etaL, 1997) 

• the Lost Opportunities study of linguistic minority adults of 1994 (Carr-Hill 
etaL, 1996) 

• an unpublished battery of test items devisedfor the Agency by NFER in 1992-93, and 

• the international Reading Literacy Study of 9-year-olds of 1991 (Elley, 1992). 
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Two main reading test forms were devised. Half the sample (randomly assigned) 
took each form at pre-test, and the other form at post-test. A complex and 
innovative statistical process was used to equate the test forms (see section E.l). 
Briefly, this involved taking the statistical values of the IALS items on the scale used 
in that study as fixed (because they were derived from a nationally representative 
sample), using those values to calculate values on the IALS scale for the non-IALS 
items, and then using the complete set of IALS items values to calculate scores on 
that scale for each student. 

The tests made literacy demands which could be considered largely functional; that 
is, they simulated public or other real-world literacy tasks likely to be faced in 
everyday life. The early sections of the reading tests made least demand on students’ 
literacy, while later sections (using IALS tasks) required them not only to read the 
stimuli but also to write their answers unaided. One-sentence writing prompts were 
also used at pre- and post-test. 

Background data were gathered on the students, plus the number of hours of tuition 
attended between pre- and post-test. Additional information was gathered by means 
of a tutor questionnaire. A very small amount of qualitative information was 
gathered from interviews with seven basic skills co-ordinators and from 
fieldworkers’ reports of serendipitous observations - they were not required, and did 
not have the opportunity, to observe teaching sessions systematically. 

The reading data were analysed against the scale used in IALS. The writing scripts 
were assessed according to a detailed analytic scheme. Both sets of results were then 
related to the background data, and the reading results also to some variables from 
the tutor questionnaire. 

The pre-tests were administered between October 1998 and May 1999. Meanwhile, 
post-testing began in March 1999 (because some provision ended at Easter), and 
continued until July. (For the explanation of the lengthy pre-testing period, see 
section A. 7.) The shortest period between pre- and post-test for any student was one 
month - in the intensive courses. In theory, the longest period between tests could 
have been nine months. In practice, because those who were pre-tested first were 
also post-tested first, the maximum period was six months. Because of the 
Christmas, Easter and half-term holidays, no student could have attended more 
than 20 weeks of provision between tests. 

A total of 2,135 students (representing provision in 71 FE Colleges and T.F.A s) took 
the reading pre-test, and 1,224 also took the reading post-test (57 per cent 
retention). Writing scripts were received from 1,724 students at ^re-test, and 937 at 
post-test (54 per cent retention). Because of the relatively low retention rates, 
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detailed comparisons were carried out between the students who returned at post- 
test and the whole pre-test sample (see section A.9). Briefly, the background 
characteristics (gender, age, occupational status, etc.) of the full sample and the 
returners’ subsample were compared; also the distributions of pre-test scores for the 
two groups. These comparisons showed that the returners’ characteristics and 
scores were a very close match for those of the full pre-test sample, and therefore 
that the returners could be treated as fully representative of all those who had taken 
the pre-test, in terms of the characteristics on which information was available. 

1.4 Structure of this report 

Descriptions of various aspects of the methodology fill the Appendices. Previous 
national studies of adult literacy in Britain and the United States are analysed in 
chapter 2. Chapter 3 provides a sketch of adult literacy teachers and teaching in 
England and Wales, and the sample of students who participated in this study is 
described in chapter 4. Chapter 5 provides the quantitative evidence on students’ 
progress in reading and writing, and in chapter 6 a search is made for factors 
associated with that progress. Chapter 7 presents an overall summary and 
recommendations. 
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Chapter 2 

Previous national studies of progress 
in adult literacy 

2.1 Previous national studies of progress in adult literacy in Britain 

There have, been three previous supposedly national studies of progress in adult 
literacy in England and Wales. The first two were carried out in the late 1970s during 
the ‘Right to Read’ campaign, best known for the associated BBC television series ‘On 
the Move’. That was also a period of very rapid expansion in adult basic education. 
The third, in 1989/90, was an attempt to study different styles of provision. 



2.1.1 Jones and Charnley (1978) 

The slightly earlier of the two studies in the 1970s (1975-77) was a qualitative survey 
of the views of organisers, tutors and students. No exact figures were given in the 
report for the numbers of people interviewed, but the number of research areas was six 
(Jones and Charnley 1978, p.9) and at a later point (p.73) the numbers of students 
interviewed in two of these areas appear to be given as 41 and 39; perhaps about 240 
students took part in all. No attempt was made to measure students’ progress directly, 
and the resulting report had little to say about progress in literacy. Jones and Charnley 
(1978, pp.93-95) provided nearly three, pages of quotations from interviews with 
students testifying to their progress, and then (p.96) gave this summary: 

[R] datively few students were found who were ready to exercise their skills in public. 
They might sign a cheque in public with confidence , but only after the details had been 
filled in at home. This reluctance was most noticeable in writing. Students generally 
believed that they had made more progress with reading than with writing; but that 
most progress of all had been in feeling better about themselves. And this was seen 
more frequently in those who had attended groups than among those under tuition at 
home. 

Later (p.l 10) they made this judgement based on all the evidence in the survey: 

The main body of those who persevered in tuition made progress in their skills that was 
slow but steady , and there were a very few examples of astonishingly rapid progress 
towards reading and writing. 
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2.1.2 GormanandMoss (1979) 

The second study in the 1970s (1976-79) was complementary to the first because it 
was a specifically and purely quantitative survey It was carried out by the NFER for 
the Department of Education and Science, and was directed by Tom Gorman (1981; 
Gorman and Moss, 1979), who also devised the scheme for the assessment of 
writing for the present study The 1976-79 survey materials contained tests of 
reading, writing and spelling, and the tests were administered at both stages by 
basic skills tutors. 

The survey ran into severe data-collection difficulties. The target sample was set 
at 2,000 students; pre-test materials were despatched in December 1977 to 1,831 
students; by March 1978, 1,238 sets of materials (68 per cent of those 
distributed) had been returned. Post-testing was undertaken in June 1979; 
scheme organisers were asked to distribute the post-tests to those students from the 
pre-test sample who were still receiving tuition. A total of 1,158 students (94 per 
cent of the pre-test sample) were traced; 378 (33 per cent of those traced) were no 
longer receiving tuition; and 194 (17 per cent of those traced) failed to respond; so 
that post-tests were received from 586 students (51 per cent of those traced). 
However, some students returned incomplete sets of post-test materials. The final 
numbers on whom pre-/post-test comparisons were based were therefore as shown 
in Table 2.1. 

Table 2.1: Numbers of students pre- and post-tested in 1976-79 adult literacy study in 
England 





N 


Percentage of pre-test sample 


Reading 


541 


44% 


Writing 


485 


39% 


Spelling 


440 


36% 


Overall 


562 


45°/o 



Note: The figure for spelling, and various others to be quoted here, are not given in the published article (Gorman, 
1 981), but are culled from the unpublished full research report (Gorman and Moss, 1979), of which archive copies are 
kept at NFER. 



Because not all items were common between the two occasions, a one-parameter 
Item Response Theory (IRT) model (Rasch scaling) with a mean of 50 and standard 
deviation of 10 was used to calibrate all items on a common scale via the items that 
were common. This was believed at the time to be a statistically valid method, and it 
was used in national surveys of schoolchildren’s literacy and numeracy attainments 
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from 1978 into the early 1980s. However, the Rasch model then came under such 
conceptual attack that it was abandoned (in school-level surveys it was replaced by 
multi-level modelling). 

The results are summarised in Table 2.2. Given the low retention rates and the 
dubious statistical model, they need to be taken with some caution. 



Table 2.2: Results of 1976-79 adult literacy study in England 





Average scaled scores 




Skill 


N 


Pre-test 


Post-test 


Average 

gain 


s.d. of 
gain 


z- 

score 


Proba- 

bility 


Effect 

size 


Reading 


541 


51.4 


52.9 


1.5 


6.5 


5.4 


p<0.001 


0.15 


Writing 


485 


53.7 


56.3 


2.6 


4.6 


12.4 


p<0.001 


0.26 


Spelling 


440 


54.0 


57.0 


3.0 


5.5 


11.4 


p<0.001 


0.30 


Overall 


562 


51.3 


53.4 


2.1 


4.6 


10.8 


p<0.001 


0.21 



Key: N = sample size; s.d. = standard deviation 

Sources: Tests of statistical significance and effect sizes calculated for this report; otheiwise Gorman and Moss (1979, 
Table 7, p.7.17). 

Note: The effect sizes were calculated by dividing the gain by the s.d. of the reporting scale (10). 

All four gain scores were statistically significant. From the effect sizes, the gains in 
writing and spelling appear educationally significant (greater than a quarter of a 
standard deviation), those for reading and the overall score less convincingly so. 



2.1.3 Abell (1992) 

In 1989/90 an attempt was made to relate different styles of provision in adult 
literacy (small group, drop-in, one-to-one, etc.) to effectiveness in terms of student 
progress. Questionnaires were sent to organisers, tutors and students. Tutors’ and 
students’ perceptions of the students’ progress were compared, and further 
triangulated against independent assessors’ reports ‘on progress identified in the 
writing of 50% of the students in the sample’ (Abell 1992, p.48). The report (p.7) 
claimed that 

96% of students who had completed a course or who had remained in provision 
throughout the study were firmly convinced that they had made progress. Reports 
from tutors and independent assessors supported these claims. ^This progress did not 
appear to have been dependent on the style of provision in which they participated. 
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However, these claims rested on a very small base (see especially Abell, 1992, Figure 
10, p.81). The number of questionnaires sent out initially to students in September 
1989 was 221, of which 174 (79 per cent) were returned. Of this group, 78 (45 per 
cent) returned the follow-up questionnaire in April/May 1990. Thus the ‘96% of 
students who . . . were firmly convinced that they had made progress' amounted to 
75 people. And the 50 per cent of the sample whose writing was assessed appears to 
have consisted of 39 people. The claims about progress should be taken as subjective 
and as referring only to a small group. 

The conclusion that ‘progress did not appear to have been dependent on the style of 
provision' was also open to question: the report conceded (for example on p.5) that 
different styles of provision could not reliably be distinguished on the ground, and in 
any case the sample of students was much too small to have supported conclusions 
about different styles, even if they could have been distinguished. It would be safer to 
conclude that this research provided no evidence either way on the relationship 
between styles of provision and progress. 

2.2 Previous national studies of progress in adult literacy in the 
United States 

Much more research has been done in the USA than in Britain on adult literacy, 
including three supposedly national progress studies. However, all of those studies 
appear problematic (the following summaries are based on the review by Beder, 1999): 

1. A 1965-68 study sampled courses in small areas of New York, New Jersey and 
California only and was therefore not genuinely national. The reading test used 
(in the absence then of instruments designed for adults) was a school-level test of 
doubtful validity for adults. The first and second post-tests were conducted six 
and 12 months after the pre-test. The numbers of people on whom post-test data 
were gathered were 1,641 and 1,425 respectively, but the number pre-tested was 
not stated. Adults who had been pre-tested but had not attended classes were 
treated as a control group - but their refusal to attend means that they were not 
a genuine control or comparison group. Both they and participants made 
equivalent gains, and the participants’ gain was therefore statistically non- 
significant- but the comparison seems unreliable. 

2. A 1972 study used a reading comprehension test designed for adults, which 
appears to have had a ceiling effect. The pre-test was in Februa^-March 1972 
and the post-test in May 1972. Of the 1,108 students pre-tested, only 441 (40 per 
cent) returned at post-test. The gains expressed in fractions of a grade level 
(roughly equivalent to months of reading age) were half a year or less, but were 
statistically significant. However, the data appear unreliable because of the ceiling 
effect, the poor retention rate and the small sample size relative to the population. 
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3. The National Evaluation of Adult Education Programs was carried out in 
1990-94. It suffered massive data-collection problems, mainly through attrition, 
and of such severity that the US government called in a new group of consultants 
to re-analyse the data. The target pre-test sample was 19,796 adults, but the 
achieved pre-test sample was 11,354 (57 per cent of the target). Post-test data 
were obtained from 2,333 people (21 per cent of those pre-tested, 12 per cent of 
the target sample). But 1,719 post-tests had to be withdrawn from analysis 
because of non-completion or floor or ceiling effects, so that gains were 
calculated on only 614 cases (five per cent of those pre-tested, three per cent of 
the target sample). The new group of consultants quite reasonably concluded 
that the test data were unusable. 

Beder’s (1999, p.5) conclusion (from these national surveys and several local US 
progress studies) was that, As measured by tests, the evidence is insufficient to 
determine whether or not participants in adult . . . education gain in basic skills/ 

A similar conclusion was reached by Sheehan-Holt and Smith (2000). They re- 
analysed data from the (US) National Adult Literacy Survey (NALS) of the early 
1990s (Kirsch et al f 1993) in which a sample of 24,944 people representative of the 
full ability range participated. This was not a progress study but a single-occasion 
‘snapshot’. Nevertheless, by appropriate statistical methods, Sheehan-Holt and 
Smith (2000, p.227) were able to compare adults ‘at the lower end of the literacy 
skills continuum’ within the survey who had participated in adult literacy 
provision, with directly comparable people who had not. The main finding was that 
‘no association was found between participation in (basic skills) programs and 
literacy skills’ (p.227). 

However, there was a positive association between participation in basic skills and 
several types of reading practice: ‘Participants read more magazines, books, 
personal documents and document materials for work than nonparticipants’ 
(p.240). Sheehan-Holt and Smith went on to discuss the plausible theory that 
increased reading would over time lead to improved reading attainment; their 
analysis revealed, however, that there were ‘no differences in literacy proficiencies 
between those who participated in basic skills programs more than one year prior 
to the NALS assessment and those who were enrolled within one year of the 
NALS’ (p.240). 

Overall, therefore, in the United States convincing national evidence has yet to be 
found, that participants in adult basic skills provision improve their literacy 
attainment. Beder (1999) reached the same conclusion from several local US 
studies also, though Sticht and Armstrong (1994) did find that other local US 
studies showed a positive impact. * 
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2.3 Summary 

/ 

• There has been only one previous quantitative national study of progress in adult 
literacy in Britain. That study was conducted over 20 years ago, its samples were 
small, and the statistical model used has since been found wanting. If reliable, the 
results suggest that reasonable progress was made in the forms of adult literacy 
provision then in use. 

• None of the national progress studies of adult literacy in the USA provided 
reliable data, while a recent re-analysis of data from a large national one-off 
survey in the USA found no evidence of benefit to literacy attainment of 
participation in adult basic skills provision. 

• It therefore appears that the study reported in this volume is the first in the 
English-speaking world to provide reliable evidence of progress in adult literacy 
based on an adequately representative national sample . 




Chapter 3 



Adult literacy teachers and teaching in 
England and Wales 

3.1 Introduction 

The Moser Report (1999, pp.57-60) provided a sketch of effective teachers 
and effective teaching of basic skills in England, based on the evidence then 
available. This chapter provides a more detailed picture for England and Wales 
based largely on quantitative findings from the tutor questionnaire (reproduced 
in Appendix D), with a few insights from fieldworkers’ impressions. The 
questionnaires were sent out and returned during the summer term of 1999. 
A total of 263 questionnaires were sent out, and 177 (67 per cent) were 
returned. 

Most of the questionnaire addressed the tutors’ adult basic skills teaching in general, 
but some sections focused on the group that had been tested by an NFER fieldworker 
- where questions focused on this group this is stated. The main topics covered were: 
the tutors’ backgrounds, their current basic skills teaching in general and current 
adult literacy teaching in particular, any classroom assistance they received, 
management support, planning, teaching, assessment, and professional 
development. 

Almost all the questions were answered by high percentages of respondents; 
where this was the case, the responses were taken to be representative of the whole 
sample, percentages were calculated on the basis of those responding, and no 
attention is drawn to the level of non-response. But where non-response was high, 
this is stated and percentages (including those for non-response) were usually 
calculated on the basis of the whole sample. All sample sizes quoted are numbers of 
tutors. 

In chapter 6 some of the factors analysed here are related to students’ progress in 
reading. 

3.2 Tutors’ background 

3.2.1 Gender 

The great majority of the tutors (90 per cent) were women^ 
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3.2.2 Qualifications 

The range and frequency of the tutors' qualifications are shown in Table 3.1. 
Table 3.1: Tutors' formal educational qualifications 



Qualification 


N 


% 


Master's degree or Doctorate 


17 


10% 


Bachelor's degree 


31 


18% ' 


Bachelor's degree which includes Education 


68 


38% 


PGCE 


29 


16% 


Teacher's Certificate 


96 


54% 


City & Guilds Certificate 9282 


86 


49% 


City & Guilds Certificate 9283 


50 


28% 


City & Guilds Certificate 9284 


10 


6% 


City & Guilds Certificate 9285 


68 


38% 


A Post experience diploma in adult literacy/basic skills 

V 


38 


22% 


A Post experience diploma in any other subject 


26 


15% 



Note: Percentages are based on the full responding sample of 177. They do not total to 100 because almost all tutors 
had more than one qualification. 



When subcategories were combined, it was found that just over half (56 per cent) of 
the tutors were graduates, 75 per cent had qualified teacher status (PGCE, Teacher’s 
Certificate, or Bachelor’s degree which included Education), and over 80 per cent 
had a qualification in teaching basic skills (mainly City and Guilds certificates). Thus 
the great majority of tutors had both OTS and a basic skills certificate. 

Where tutors had a first or higher degree, they were asked to state the main 
subject(s). The major categories were English and Education, with Humanities 
(broadly defined) third, and very few reporting mathematics or science - but these 
were adult literacy tutors. 



3.2.3 Teaching experience 

Tutors were asked how many years’ teaching experience they had had up to the end 
of the academic year 1998/99, in total and in adult literacy They were asked to give 
figures separately for full-time teaching (over 35 hours a week) and part-time 
teaching. The results are shown in Table 3.2. 
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Table 3.2: Tutors' teaching experience, in years 





Overall 


In adult literacy 


Full-time 


Part-time 


Full-time 


Part-time 


Years 


N 


% 


N 


% 


N 


% 


N 


% 


0-5 


36 


44% 


46 


38% 


30 


53% 


73 


47% 


6-10 


16 


19% 


40 


33% 


16 


28% 


41 


26% 


11-15 


20 


25% 


19 


16% 


8 


14% 


24 


14% 


16-20 


4 


5% 


10 


8% 


2 


4% 


10 


8% 


20+ 


6 


7% 


7 


6% 


1 


2% 


6 


4% 


Total 


82 


100% 


122 


100% 


57 


100% 


156 


100% 



Note: The percentages relate to the totals for the relevant categories. 

A total of 82 tutors (46 per cent of the total) had full-time teaching experience; of 
these, nearly half had less than six years’ experience and only just over one third 
had more than 10 years’ experience. Fifty-seven tutors had full-time experience in 
adult literacy, of whom just over half had less than six years’ experience and only a 
fifth had more than 10 years’ experience. More tutors reported having part-time 
adult literacy teaching experience than reported having part-time teaching 
experience in general; this may imply that some had taught adult literacy part-time 
while holding a full-time teaching post and/or that some tutors’ only teaching 
experience was part-time in adult literacy 



3.3 The tutors’ current basic skills teaching in general 

3. 3. 1 Working hours , overall 

The tutors were asked whether they were currently teaching adult basic skills full- 
time or part-time: 50 (28 per cent) were teaching full-time and 125 (71 per cent) 
part-time (two did not answer this question). 

The Basic Skills Agency’s statistical report for 1996/97 (BSA. 1999, Table 10, p.9) 
provided national data from which the proportion of paid adult literacy and 
numeracy staff in England and Wales who were full-time in that year could be 
calculated, namely 9.3 per cent, or three times the proportion in this study. 
However, paid staff’ in the BSA’s statistics included paid assistants as well as tutors; 
and it seems likely that full-time tutors would have been mcjfre likely to respond to 
the questionnaire used in this study. * 
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The Moser Report (1999, pp.29, 58) drew attention to the largely part-time nature 
of the basic skills teaching force, and called for the proportion of full-time staff to be 
increased. 

Tutors were also asked about the number of hours per week they were teaching 
basic skills, both in total and, within that, adult literacy. ‘Teaching’ was defined as 
contact hours with students, including resource-based learning. The results are 
shown in Table 3.3. 



Table 3.3: Tutors’ teaching hours, overall and in adult literacy 



Contact hours per week 


Overall 


Adult literacy 




N 


% 


N 


% 


<5 


18 


10% 


29 


17% 


5-9 


38 


22% 


54 


32% 


10-14 


34 


20% 


43 


25% 


15-19 


40 


23% 


23 


13% 


20-24 


33 


19% 


15 


9% 


25-29 


4 


2% 


2 


1% 


30-34 


5 


3% 


2 


1% 


>34 


3 


2% 


3 


2% 


Total 


175 


100% 


171 


100% 



Note: The percentages relate to the totals for the relevant categories. 

Just over half the respondents were teaching less than 15 hours per week overall. 
Very few (12, or seven per cent) were teaching more than 24 hours a week; this 
implies that almost all those working full-time had non-contact time for 
preparation, assessment, co-ordination and administration. Almost half the 
respondents were teaching less than 10 hours a week of adult literacy. The 
distribution of hours teaching adult literacy against overall teaching hours, implies 
that most tutors spent about one-third of their hours teaching subjects other than 
adult literacy (perhaps mainly numeracy, though this question was not asked). 



3.3.2 Hours teaching tested group 

More specifically, tutors were asked to state the number of hours per week they 
taught the group that was visited and tested by an NFER lieldworker; the results are 
shown in Table 3.4. 
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Table 3.4: Numbers of hours per week tutors taught provision visited by NFER fieldworker 



Hours per week 


N 


% 


2 


91 


55% 


3 


29 


17% 


4 


18 


11% 


5 


3 


2% 


6 


14 


8% 


7-24 


12 


7% 


Total 


167 


100% 



3. 3. 3 Size of teaching groups 

Tutors were asked to state the average size of the adult literacy groups they taught, 
in terms of the number of students usually present (not the number on roll). The 
results are shown in Table 3.5. 



Table 3.5: Average size of adult literacy groups taught 



Size of group 


N 


% 


1-5 


19 


11% 


6 


30 


17% 


7 


19 


11% 


8 


57 


33% 


9 


10 


6% 


10 


25 


15% 


11-30 


13 


8% 


Total 


173 


100% 



Just over half taught the relevant group two hours per week, with a further 38 per 
cent teaching that group between three and six hours a week. The small number 
reporting larger numbers of hours per week were probably those running drop-in 
and/or intensive provision. 

Four-fifths of respondents reported group sizes of between'six and ten. The few very 
large group sizes may represent tutors supervising dfopan provision and/or those 



0 



Adult literacy teachers and teaching in England and Wales 2 3 



who taught more than one group and perhaps gave a total rather than an average. 
The very small groups probably represented both some one-to-one tuition and the 
fact that the data were collected late in the academic year. 

The general picture so far confirms statistically what is known informally: this 
is a largely part-time and very largely female profession, predominantly qualified 
in Arts subjects and predominantly teaching small groups a few hours per 
week. 



3.4 Professional development 

Tutors were asked if they had attended any professional development (in-service) 
activities (such as courses, conferences, seminars) in the three academic years since 
September 1996: 156 (91 per cent of those responding) said they had. The areas 
covered are shown in Table 3.6; the range mainly covered practically relevant 
topics. 



Table 3.6: Areas covered in professional development, 1996-99 



Area 


N 


% 


Teaching of reading 


74 


42% 


Teaching of writing 


68 


38% 


Recent research in literacy 


78 


44% 


Phonological awareness 


38 


22% 


IT and literacy 


69 


39% 


Other 


78 


44% 



Note: The percentages shown are based on the full sample of 177, and do not sum to 100 because tutors could give 
more than one answer. 



There were also several indirect questions concerning tutors’ professional 
know ledge. They were asked to name the book on the teaching of literacy they 
had read most recently, and state the year in which they read it. A total of 
141 tutors (80 per cent of the total) gave a title. The titles were too numerous to 
list, and only three received more than four mentions each, so Table 3.7 
instead shows broad categories. Some titles were forms of publication other than 
books. 
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Table 37: Categories of publication on the teaching of literacy read most recently 



Category 


Number of mentions 


ALBSU/BSA publications 


42 


Books on dyslexia 


35 


Books on spelling 


13 


General adult learning 


8 


Educational press 


5 


Moser Report 


4 


Books on EAL 


2 


Other 


32 


Total 


141 



The Moser Report was hot off the press when the questionnaires went out; it is 
therefore interesting that even four tutors had read it. The categories showed a 
heavy emphasis on practical information directly relevant to students' special 
educational needs. Fifty-seven tutors (32 per cent of the sample) did not state the 
year in which the book was read. Of the 120 who did, 91 (76 per cent) said 1999, 
22 (18 per cent) said 1998, and only seven (6 per cent) gave an earlier date. 

Tutors were also given a list of 14 (mainly British) authors who have published 
widely on literacy and asked to indicate how familiar each author's name was to 
them. Non-responses ranged between 25 and 37 per cent. Because ‘Not familiar' 
was the response of at least half of respondents to every name, figures for ‘familiar’ 
and ‘some knowledge’ were combined. The resulting data for familiarity are shown 
in Table 3.8 overleaf in descending order. 

The list of authors was very mixed. In particular no author on it could be considered 
to have written the equivalent for adult literacy of Roger Beard’s analysis of the 
evidence base for the National Literacy Strategy - because so far no-one had written 
such an analysis. However, the responses yielded some interesting findings. 

The two ‘gurus’ of the whole language movement, Ken Goodman and Frank Smith, 
were the most familiar names - but their reputations were made in the 1960s and 
early 1970s. They were also the only two non -British authors on the list. The two 
authors whose work has been most concerned with adult literacy, David Barton and 
Mary Hamilton, were in middling positions. It is perhaps more understandable that 
the three whose work has been most concerned with details of initial teaching (Sue 
Lloyd, Henrietta Dombey, Liz Waterland) should have been least familiar. 
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Table 3.8: Tutors' familiarity with particular authors' work on literacy 



Author 


Familiar 




N 


% 


K. Goodman 


47 


27% 


F. Smith 


44 


25% 


M. Snowling 


37 


21% 


R. Beard 


35 


20% 


P. Bryant 


32 


18% 


L. Bradley 


31 


18% 


D. Barton 


29 


16% 


U. Goswami 


24 


14% 


N. Bielby 


20 


11% 


M. Hamilton 


20 


11% 


M. Meek Spencer 


16 


9% 


S. Lloyd 


10 


6% 


H. Dombey 


10 


6% 


L. Waterland 


9 


5% 



Note: The percentages are based on the full sample of 177. 



The overall impression from the information in this section is that the tutors 
who responded did largely keep up to date in terms of attending courses and 
reading books relevant to students’ needs, but the authors who were most 
familiar to them had produced their most influential work some time ago. An 
implication for in-service training might be that subject knowledge of the 
practicalities of literacy teaching needs to be embedded in a broader context of up- 
to-date theory. 



3.5 Management support 

Tutors were asked what types of support they received from their line managers: 1 3 
(seven per cent) said no support was provided. The results for forms of support 
which were reported by the other tutors are shown in Table 3.9. 
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Table 3.9: Types of support received by tutors from line managers 



Type of support 


N 


% 


Out of hours support 


98 


60% 


Visits to their class 


80 


49% 


Drop-in provision for tutors 


60 


37% 


Enabling them to observe other tutors 


33 


20% 


Other 


35 


21% 


Total 


164 


100% 



Note: Numbers and percentages do not total to 164 and 100 respectively because tutors could give more than one 
answer. 



Among ‘other' forms of support, 20 tutors (11 per cent) said they received constant 
support from their line managers, or could always refer to or contact them, and 1 1 
tutors (six per cent) mentioned meetings (see section 3.6) as a form of support. 

Overall, management support for tutors appeared patchy. This supports the 
comment in the Moser Report (1999, p.29) that the very part-time nature of the 
teaching force in adult basic skills ‘necessarily restricts the amount of continuous 
professional training that teachers are able or willing to go to. The fact that 
basic skills teaching remains an area where there are few career opportunities 
for professional development means that it still remains an unattractive 
career . . . (I)t . . . makes it more difficult to recruit new teachers.’ 



3.6 Assistance with teaching 

Tutors were asked if they regularly had volunteers and/or paid assistants working 
with them in the group visited for this study and, if so, how frequently this occurred, 
how many helpers were present on average per session, what aspects of the work the 
volunteers and assistants helped with, and how they (the tutors) organised this. 

A total of 110 tutors (62 per cent) said they had people working with them, of 
whom 96 had volunteers and 22 had paid assistants (so that a few had both). The 
average number of volunteers present per session was one in 55 cases, two in 29 
cases, and three or more in the remaining 12 cases - the largest number stated was 
nine (perhaps by a tutor running drop-in provision). One of the 22 tutors who had 
paid assistants had two, the rest having one each. 
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The figures for the frequency of this assistance are summarised in Table 3.10. The 
great majority of the tutors who had assistance had it most of the time - but it 
should be remembered that 67 tutors (38 per cent) said they had no assistance at all 
with the group visited for this study 



Table 3.10: Average frequency over a term of assistance in class 



Frequency 


N 


% 


Almost even/ session 


85 


77% 


About l U °f the sessions 


18 


16% 


Less than l U of the sessions 


7 


7% 


Total 


no 


100% 



The aspects of the work which the volunteers and assistants helped with are shown 
in Table 3.11. 

Table 3.11: Aspects of work supported by helpers 



Aspect of the work 


Volunteers 


Paid assistants 


N 


% 


N 


% 


Support for individual students 


84 


88% 


22 


100% 


Support for more basic students 


81 


84% 


15 


■ 68% 


Support for whole group 


30 


31% 


10 


45% 


Planning 


16 


17% 


7 


32% 


Total 


96 


100% 


22 


100% 



Notes: The Ns shown are numbers of tutors reporting that form of assistance from volunteers/paid assistants. The 
percentages are based on the total numbers of tutors saying they had volunteers/paid assistants. The percentages for 
paid assistants are based on a small total and should be treated with caution. Numbers and percentages do not total to 
96/22 and 100 respectively because tutors could give more than one answer. 

Thus the great majority of the assistants’ efforts went, as might be expected, into 
supporting individual students, especially those with the greatest need. Many of the 
tutors’ answers to a question on how they organised the work of their assistants, 
repeated this theme of giving individual support as and where needed. 

However, 104 tutors (59 per cent of the total) gave answers to the organisation 
question which dealt with their management of their helpers. The predominant 
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themes were that the tutor and the assistant, sometimes with the student, discussed, 
devised and adapted the ‘study programme’. 

Overall it seemed that appropriate use was being made of assistants, where they 
were available. No explanation was available of why nearly two fifths of tutors had 
no assistance. The importance of this variable is underlined by the fact that it was 
the only ‘teaching’ variable which was found to have a positive relationship with 
students' progress (see section 6.4.). 



3.7 Planning 

3. 7. 1 Meetings 

Two questions asked how frequently tutors had formal or informal meetings with 
other tutors to discuss and/or plan their adult literacy curriculum or teaching 
approaches in general; the results are shown in Table 3.12. 



Table 3.12: Frequency of formal and informal meetings 



Frequency 


Formal 


Informal 


N 


% 


N 


% 


Every week 


5 


3% 


54 


31% 


More than twice a term but not weekly 


29 


16% 


61 


35% 


Twice a term 


36 


20% 


9 


5% 


Once a term 


87 


49% 


24 


14% 


Never 


11 


6°/o 


21 


12% 


Total 


168 


100% 


169 


100% 



That even a few tutors reported never having a formal meeting seems 
unsatisfactory, but perhaps the very short hours worked by many tutors made it 
impossible for them to attend, or for meetings to be arranged for them. The great 
majority had at least one formal meeting a term, and informal meetings were 
(understandably) considerably more frequent. 



3.7.2 Curricula and schemes of work 

Tutors were next asked if they followed a defined curriculum and (in a further 
question) a scheme of work in planning their adult literacy teaching; 88 (50 per 
cent) said they followed a defined curriculum and 79 (45 per cent) said they did not 
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(10 did not answer this question), while 147 (83 per cent) said they followed a 
scheme of work and 24 (14 per cent) said they did not (six did not answer this 
question). 

Those who said they did follow a defined curriculum or a scheme of work were then 
asked about the basis of the curriculum and scheme of work; the results are shown 
in Table 3.13. 

Table 3.13; Basis of tutors' adult literacy curricula and schemes of work 



Basis 


curriculum 


scheme of work 


N 


% 


N 


% 


based on an accreditation framework 


97 


55% 


122 


69% 


devised by tutor 


59 


33% 


114 


64% 


involving the students 


16 


9% 


24 


14% 


devised by tutor's Programme Manager 


9 


5% 


17 


10% 


Total 


167 


100% 


171 


100% 



Notes: Numbers and percentages do not total to 167/171 and 100 respectively because tutors could give more than 
one answer. The fact that the percentages for curriculum sum to 102 is coincidental. 



More tutors answered these questions than said they followed a defined curriculum 
or a scheme of work, and the data in Table 3.13 should therefore be taken to refer to 
curricula/schemes of work in general - this is also implied by the proportions of 
tutors who said they devised their own. Both involving students and relying on 
the Programme Manager were infrequent responses - most tutors derived their 
curricula and schemes of work from an accreditation framework or devised them 
themselves, and the data imply that many derived their schemes of work at least 
from both sources. 



3.7.3 Sources of information for planning 

More detail on tutors’ planning of their adult literacy provision was gained from a 
multi-part question asking how much they relied on various sources of information. 
The results are shown in Table 3.14 in decreasing order of the response A great 
deaf. 

By far the most commonly relied upon sources were students’ individual learning 
plans and feedback from tutors’ observations and assessments of students’ work, in 
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other words evidence readily available in the classroom. Even their own previously 
prepared classes were relied upon by far fewer tutors. Textbooks seemed 
insignificant as a source of planning for most tutors, despite the substantial reading 
of BSA publications reported in Table 3.7 above and the substantial use of such 
materials reported in section 3.8.2 below. The very low percentage of tutors relying 
a great deal on other tutors or ABS specialists in their LEA/ College seems consistent 
with the largely part-time nature of the teaching force, and with the low frequency 
of meetings already mentioned. 



Table 3.14: Sources relied on in planning adult literacy provision 



Source 


A great deaf 


A little 


Not at all 


TOTAL 


N 


% 


N 


% 


N 


o/o 


N 


% 


students' individual learning plans 


157 


91% 


16 


9% 


0 


0% 


173 


1 00% 


feedback from their observations 
and assessments of students' work 


150 


88% 


18 


11% 


2 


1% 


170 


1 00% 


their own previously prepared 
classes 


49 


30% 


101 


62% 


13 


8% 


163 


100% 


textbooks or resource books other 
than those mentioned below 


34 


21% 


123 


74% 


9 


5% 


166 


1 00% 


external assessments 


25 


16% 


82 


52% 


52 


330/o 


159 


1 00% 


other tutors or ABS specialists in 
their LEA/Co liege 


15 


9% 


89 


56% 


55 


350/o 


159 


1 00% 


tutors' guides or tutor edition of 
textbook 


14 


9% 


89 


57% 


52 


340/o 


155 


100% 


students' edition of textbooks 


9 


6% 


70 


47% 


69 


470/o 


148 


100% 



Note: The Total Ns and percentages are row totals, and the percentages in each row are based on the total N for that row. 



3. 7.4 Planning students’ progression in learning 

Tutors were next asked how much weight they gave to various forms of evidence on 
their students’ progress when planning their students’ further progression in 
learning. The results are shown in Table 3.15, in descending order of the response 
A great deal’. 
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Table 3.15: Weight given by tutors to various forms of evidence in planning their students' 
progression in learning 



Form of evidence 


Weight given in planning students' learning 


A great deal 


Some 


None 


Total 


N 


% 


N 


% 


N 


0/0 


N 


% 


Continuous assessment 


140 


81% 


31 


18% 


1 


1% 


172 


1 00% 


Student self-assessment 
negotiated with the tutor 


124 


72% 


45 


26% 


3 


20/o 


172 


100% 


Students' portfolios 


83 


49% 


81 


48% 


5 


3% 


169 


1 00% 


How well students do on written 
assignments 


79 


46% 


90 


53% 


2 


1% 


171 


.100% 


How well students do in oral work 


49 


29% 


112 


66% 


9 


5% 


170 


1 00% 


Tutor-made tests 


32 


19% 


89 


51% 


52 


30% 


173 


100% 


Listening to students read aloud 


24 


14% 


103 


60% 


46 


27% 


173 


100% 


Published tests 


16 


90/o 


75 


44% 


80 


47% 


171 


100% 



Note: The Total Ns and percentages are row totals, and the percentages in each row are based on the total N for that row. 



The categories listed in Table 3.15 were far from clear-cut; in fact all but the two test 
categories were aspects of continuous assessment. It is therefore not surprising that the 
great majority of tutors gave continuous assessment most weight. Although just over 
two fifths of respondents gave no weight to published tests when planning their 
students’ progression in learning, and about three in ten gave no weight to tests they 
devised, still a majority gave some or a great deal of weight to these forms of evidence. 
Thus again the overall picture was of more reliance on sources of evidence which are 
readily and immediately available in the classroom. 

The overall impression of planning for teaching was one of flexibility and individual 
initiative, with relatively little reliance on colleagues and substantial reliance by tutors 
on their own resources and on readily available evidence. To some, this may reflect a 
high level of professionalism; to others, it may seem like ‘winging it’ - but this 
approach may be largely forced on adult literacy tutors by the very part-time nature of 
their work, and by their students’ wide range of attainment (see chapters 4 and 5) and 
mainly intermittent pattern of attendance. 

3 . 8 Teaching and learning in general 

3. 8. 1 Modes of teaching 

Tutors were next asked to say approximately what proportion of the time in their 
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adult literacy classes was occupied by each of four modes of teaching; the results are 
shown in Table 3.16. Because non-response was unusually high for three of the four 
modes, non-response has been shown as a separate category so that percentages 
are comparable for all four modes. 

Individual teaching was the most frequent mode, and teaching groups selected by 
ability the least - two fifths of respondents never taught their students in such 
groups. Nearly one third of tutors taught individually all the time; the rest appeared 
to blend this mode with class teaching and teaching of small mixed-ability groups, 
with some also using ability grouping occasionally. 



Table 3.16; Approximate proportion of time in adult literacy classes occupied by different 
modes of teaching 





Mode of teaching 


Individual 


Class 


Mixed-ability 


Ability groups 


Proportion of time occupied by 
mode of teaching 


N 


% 


N 


% 


N 


% 


N 


% 


All 


54 


31% 


6 


30/0 • 


19 


110/0 


9 


50/0 


about 3 / 4 


40 


23% 


4 


20/o 


4 


2o/o 


5 


30/0 


about y 2 


38 


22% 


34 


19% 


15 


9% 


13 


70/0 


about V 4 


35 


20% 


56 


320/o 


60 


340/o 


42 


240/o 


(Almost) never 


1 


10/0 


51 


29% 


40 


23% 


.70 


40% 


No response 


9 


50/o 


26 


15% 


39 


220/o 


38 


220/o 


Total 


177 


100% 


177 


100% 


177 


100% 


177 


IOO 0/0 



3.8.2 Content of schemes of work 

Tutors were asked how often they included certain items in their scheme of work. 
The results are shown in Table 3.17, in decreasing order of the response ‘Every 
lesson’. 

There seemed to be a fairly clear division between the first six items in the list and 
the rest. The first six had combined ‘Every lesson’ and ‘Most lessons’ frequencies of 
between 58 and 85 per cent, while the rest had combined frequencies of 41 per cent 
or less. While some of the other items would no doubt be more difficult to include 
more frequently, the first six items together give a picture of tutors focusing 
predominantly on students’ writing skills, in the context^pf writing for their own 
purposes, and on their ability to read and reason. The other items on the list, though 
focusing on rather more mechanical skills, were also covered. 
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Table 3.17: Frequency of inclusion of items in scheme of work 



Item 


Every lesson 


Most lessons 


Some lessons 


Never/almost 

never 


Total 


N 


o/o 


N 


% 


N 


0/0 


N 


% 


N 


% 


Work on spelling 


62 


36% 


84 


49 % 


23 


130/o 


3 


2 % 


172 


1 00 % 


Work to develop 
spelling strategies 


45 


270/o 


77 


46% 


43 


250/o 


1 


lo/o , 


166 


100 % 


Students writing for 
their own purposes 


38 


22 % 


73 


42% 


56 


320/o 


2 


lo/o 


169 


1 00 % 


Work to develop 
reading strategies 


33 


19% 


70 


41% 


58 


34 % 


7 


4 % 


168 


1 00 % 


Explaining the 
reasoning behind 
an answer 


33 


19o/o 


68 


39% 


68 


39 % 


2 


lo/o 


171 


100 % 


Work on punctuation 


29 


■ 170/o 


82 


48% 


58 ! 


340/o 


2 


lo/o 


171 


100 % 


Work to develop a 
sight vocabulary 


16 


10 % 


30 


18% 


93 


56% 


24 


15% 


163 


1 00 % 


Work on whole 

language/language 

experience 


15 


9% 


55 


320/o 


91 


530/o 


10 


6 % 


171 


100 % 


Work on hand- 
writing skills 


14 


80/0 


28 


16% 


105 


61% 


24 


14% 


171 


1 00 % 


Reading a passage 
and answering 
questions on it 


14 


80/0 


57 


330/o 


95 


55% 


5 


30/0 


171 


100 % 


Work on phonics, 
including phonemic 
awareness 


12 


70/0 


47 


28% 


96 


56% 


14 


8 % 


169 


100 % 


Reading aloud 


10 


6 % 


30 


17% 


94 


55o/o 


38 


220/0 


172 


1 00 % 


Grammar/syntax 


10 


6 % 


53 


31% 


97 


570/o 


9 


5% 


169 


1 00 % 


Cloze exercises 


3 


20/0 


27 


16% 


114 


66 % 


28 


16% 


169 


100 % 


Matching exercises 


4 


2 % 


14 


8 % 


114 


67% 


36 


210/0 


168 


100 % 



Note: The Total Ns and percentages are row totals, and the percentages in each row are based on the total N for that row. 



3.8.3 Factors contributing to students' progress 

Tutors were asked, ‘To make progress in literacy, how important do you think it is 
for students to . . . ’, followed by a list of possible contributory factors. The results are 
shown in Table 3.18 in decreasing order of the response ‘Very important’. 
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Table 3. 1 8: Perceived importance of various factors in making progress 



Factor 


Very 

important 


Somewhat 

important 


Not 

important 


TOTAL 


N 


o/o 


N 


% 


N 


o/o 


N 


o/o 


Develop self-confidence 


169 


98 % 


4 


2% 


0 


0% 


173 


100% 


Work on topics which are 
relevant to them 


168 


97% 


6 


3% 


0 


0% 


174 


100% 


Be given support, evaluation 
and tutorials 


. 164 


95% 


7 


4% 


0 


0% 


171 


100% 


Find their teaching group safe, 
socially 


161 


93% 


12 


7% 


0 


•0% 


173 


100% 


Regularly achieve shortterm goals 


161 


93% 


12 


7% 


0 


0% 


173 


100% 


Have ownership of their own 
learning 


159 


92% 


14 


8% 


0 


0% 


173 


100% 


Understand how literacy is used 
in real world 


120 


69% 


50 


29% 


3 


2% 


173 


100% 


Read regularly 


107 


62% 


63 


36% 


3 


2% 


173 


100% 


Write in an organised way 


111 


64% 


62 


36% 


0 


0% 


173 


100% 


Develop phonological awareness 


68 


40% 


100 


59% 


3 


2% 


171 


100% 


Understand concepts about print 


53 


32% 


87 


52% 


25 


15% 


165 


100% 


Remember spellings 


48 


28% 


119 


70% 


2 


1% 


169 


100% 


Carry out home study 


36 


21% 


118 


69% 


18 


11% 


172 


100% 



Note: The Total Ns and percentages are row totals, and the percentages in each row are based on the total N for that row. 



Table 3.18 reveals a split between the first six items, rated as ‘very important' 
by over 90 per cent of respondents, and the rest. The top group of items has 
to do with general principles which are applicable to all forms of learning, 
while those rated less important overall are those which have to do directly 
with literacy (and study). Thus tutors appeared, on average, to rate the ethos 
of their students’ learning circumstances as somewhat more important 
than the specifics of content. This would seem to indicate the persistence 
among adult literacy tutors of a strong iiberal/humanist/community service’ 
ethos, with rather less emphasis on students’ progress and teaching directed 
to it. Such teaching would entail a detailed examination of the processes 
that contribute to extending students’ repertoire of reading and writing 
skills. 
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3.9 The tutors’ current adult literacy teaching in particular 

For all the questions analysed in this section, tutors were asked to respond in terms 
of the adult literacy provision which they taught and which had been visited by an 
NFER fieldworker. 

3.9.1 Access to IT 

Tutors were asked how frequently students had access to computers and other 
IT equipment; of the 1 74 tutors who responded, 94 (54 per cent) said their students 
had such access every session, 51 (29 per cent) some sessions, and 29 (17 per cent) 
never. 

3.9.2 Teaching materials 

Next, tutors were asked two parallel sets of three questions about teaching materials. 
The questions concerned (1) textbooks or other commercially produced materials; (2) 
materials they devised themselves. They were asked, first, whether they used/devised 
such materials for teaching adult literacy to the group visited for this study and, if so, 
to list those they used/devised most frequently, and to state approximately what 
percentage of their adult literacy teaching time they based on the materials. In all, 161 
tutors (91 per cent) said they used published materials, and 170 (96 per cent) said 
they devised their own, so the great majority used both. Results on published materials 
and the skills they address mentioned by at least 10 tutors are shown in Table 3.19, on 
types of self-devised materials mentioned by at least 10 tutors in Table 3.20, and on 
percentages of teaching time based on the two forms of material in Table 3.21. 



Table 3.19: Published materials used by tutors 



Frequently mentioned publications 


Number of tutors mentioning (>9) 


Brown and Brown 


30 


ALBSU/BSA resources 


17 


MCH resources 


14 


Alpha to Omega 


14 


Various sets of readers 


12 


Chalkface Project 


11 


Flexi-Packs 


10 


Frequently mentioned skills 


Spelling 


39 


General English skills 


14 
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Many other publications and categories of skill besides those listed in Table 3.19 
were mentioned by small numbers of tutors. Though many authors of materials 
were named, none was mentioned by more than five tutors. 

Table 3.20: Self-devised materials used by tutors 





Number of tutors mentioning (>9) 


Comprehension activities 


30 


Materials devised to meet individual students' needs 


26 


Alphabet/phonics 


25 


Materials from materials and magazines 


23 


Worksheets, general 


22 


Worksheets, spelling/punctuation 


16 


Materials for reading practice and/or modelling of writing 


15 


Punctuation activities 


13 


Cloze procedure 


12 


'Real' reading matter 


10 



Like the categories of publication read recently, analysed in section 3.3, both the published 
and the tutor-devised materials mentioned seemed to focus heavily on the practical skills of 
reading and writing. This finding seemed to be in some tension with the finding in the 
previous section, that tutors appeared on average to rate the ethos of their students’ 
learning circumstances as somewhat more important than the specifics of content. 



Table 3.21 : Percentages of teaching time based on published and tutor-devised materials 



Percentage of teaching time 
based on the material 


Type of material 


Published 


Devised by tutor 


N 


% 


N 


% 


76-100% 


4 


3% 


26 


15% 


51-75% 


21 


13% 


41 


24% 


26-50% 


54 


34% 


60 


35% 


0-25% 


82 


51% 


45 


26% 


Total 


161 


100% 


172 


100% 



Note: Percentages for published material do not sum to 100 because of rounding errors. 
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Only about one tutor in six said they based more than half their teaching time on 
published materials; the corresponding proportion for tutor-devised materials was 
about two fifths. All the tutors appeared to rely on both forms of material to some extent. 

3.10 Assessment 

3.10.1 Co-ordination of record-keeping 

A question (to which more than one response could be ticked) was put to tutors on 
how record-keeping was co-ordinated: 69 (39 per cent of the full sample of 1 77) said 
that they kept records themselves, 61 (35 per cent) that students each kept their 
own records, 59 (33 per cent) that records were kept centrally, and 48 (2 7 per cent) 
that other tutors also kept records. Practice therefore varied considerably with more 
than one system operating in some places, presumably particularly where records 
were not kept centrally and tutors each kept their own. 

3.10.2 Students 9 goals 

Tutors were asked to indicate which goals each of their students was working 
towards. About 25 tutors did not answer this question; aggregated results for the 
rest, and the categories of goal which were specified in the questionnaire, are shown 
in Table 3.22. 



Table 3.22: Numbers of students working towards different goals 



Type of goal 


Students 


Number of tutors with students 
working towards this goal 


N 


% 


Portfolio-based accreditation 


787 


56% 


151 


Self-determined goals 


701 


50% 


129 


Tests/exam-based accreditation 


293 


21% 


69 - 


None of these 


11 


1% 


8 


Total 


1411 


100% 





Note: The percentages do not total to 100 because students could be working towards more than one type of goal. 



Hardly any students were working towards none of the goals mentioned, and at 
least 28 per cent were working towards more than one type of goal. Portfolio-based 
accreditation and self-determined goals were the most frequently reported. 
Externally-determined (test- or examination-based) accreditation was reported for 
only one student in five, and the great majority of students were therefore working 
towards goals set within the provision. 




'5 
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3. 1 0. 3 Assessment of progress 

Finally, tutors were asked how much weight they gave to various forms of evidence 
when assessing their students’ progress. The items were identical to those in the 
question on planning students' progression in learning, above, and the results are 
shown in Table 3.23, in descending order of the response ‘A great deal'. 



Table 3.23: Weight given by tutors to various forms of evidence in assessment 



Form of evidence 


Weight given in assessment 


A great deal 


Some 


None 


Total 


N 


% 


N 


% 


N 


% 


N 


% 


Continuous assessment 


151 


870/o 


20 


12% 


2 


lo/o 


173 


1 00% 


Student self-assessment 
negotiated with the tutor 


113 


670/o 


54 


32% 


2 


lo/o 


169 


100% 


Students' portfolios 


108 


64% 


58 


340/o 


4 


20/o 


170 


1 00% 


How well students do on written 
assignments 


92 


46% 


90 


53% 


2 


lo/o 


171 


100% 


How well students do in oral work 


66 


39% 


97 


570/o 


7 


4% 


172 


100% 


Tutor-made tests 


43 


25% 


79 


46% 


50 


29% 


172 


100% 


Listening to students read aloud 


40 


23°/o 


85 


50% 


46 


270/o 


171 


100% 


Published tests 


27 


16% 


72 


420/o 


71 


42% 


170 


100% 



Note: The Total Ns and percentages are row totals, and the percentages in each row are based on the total N for that row. 



The pattern of responses was very similar to that for planning progression in 
learning. The great majority of tutors gave continuous assessment most weight. In 
particular, almost all gave student self-assessment and students’ portfolios, written 
assignments and oral work some or a great deal of weight for both purposes. 
Although just over two fifths of respondents gave no weight to published tests for 
assessment, and about three in ten gave no weight to tests they devised or to 
listening to students read aloud, still a majority gave some or a great deal of weight 
to these forms of evidence. Thus again the overall picture was of a blend of practices, 
with heavy reliance on easily available forms of evidence. 



3.11 Summary 

• Adult basic skills is a largely part-time and very largely female profession. Tutors 
are predominantly qualified in Arts subjects and mostlyTeach small groups a few 
hours per week. 



0 
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• Adult literacy tutors seemed to keep largely up to date in terms of attending 
courses and reading books relevant to students' needs, but the writers on literacy 
who were most familiar to them had produced their most influential work some 
time ago. 

• Overall, professional development and management support for tutors appeared 
patchy and this was consistent with one of the findings of the Moser Report. 

• It seemed that appropriate use was being made of classroom assistants, where 
they were available. No explanation was available of why nearly two fifths of 
tutors had no assistance. 

• Tutors rated the ethos of their teaching as somewhat more important than 
attention to the practical skills of reading and writing, but the published 
materials they used, and those they devised themselves, did seem to address those 
skills directly. 

• The overall impression of teaching was one of flexibility and individual initiative. 
This could be seen either as professionalism or as 'winging it'. 



Chapter 4 



What were the students like? 



This chapter provides information on the student sample based mainly on the 
Student Profile (see Appendix C), supplemented by evidence from the students' 
hours of literacy tuition between pre- and post-test, a comparison of the students' 
performance with that of a national sample of 9-year-olds, and two items from the 
tutor questionnaire. 



4. 1 Characteristics of the sample 

The Student Profile gathered information on students' gender, age, ethnicity, first 
and any other languages, occupational status and highest qualification, on whether 
they had been in basic skills tuition before the academic year 1998/99, on the 
form(s) of provision which they were attending in that year, and on the modes of 
teaching they were experiencing. Profiles were returned on 1,915 students, or 90 
per cent of those who took the reading pre-test. The information for the full sample 
is shown in Table 4.1, and details for the intensive students on two of the variables 
are given below the Table. 



Table 4.1 : Characteristics of the sample of students 





N 


% 


Total 


1915 


100 




Sex 


Men 


819 


45 


Women 


1023 


56 


Total 


1842 


100 


no information 


73 
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Age 


50+ 


280 


17 


40-49 


336 


20 


30-39 


483 


29 


20-29 


325 


19 


under 20 


248 


15 


Total 


1672 


100 


no information 


243 





Occupational status 


Full-time employed 


265 


15 


Part-time employed 


215 


12 


Full-time self-employed 


16 


1 


Part-time self-employed 


15 


1 


Unemployed 


465 


26 


Full-time education 


174 


10 


Temporarily sick/disabled 


52 


3 


Permanently sick/disabled 


149 


8 


Looking after home/family 


288 


16 


Other 


163 


9 


Total 


1802 


100 


no information 


113 






42 Progress in Adult Literacy 



Highest qualification 


None 


940 


50 


A Basic Skills certificate 


124 


7 


CSE/GCSE 


269 


14 


O-Level 


29 


2 


FE 


475 


25 


A-Level 


16 


1 


Higher education 


38 


2 


Total 


1891 


100 


no information 


24 





Ethnicity 


White 


1556 


84 


Black - Caribbean 


82 


4 


Black - African 


22 


1 


Black - Other 


7 


<1 


Indian 


28 


2 


Pakistani 


57 


3 


Bangladeshi 


14 


1 


Chinese 


17 


1 


Other 


79 


4 


Total 


1862 


100 


no information 


53 
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First language 


English 


1510 


82 


Other 


334 


18 


Total 


1844 


100 


no information 


71 





Languages Other than English (Percentages not given because of small numbers) 


Sign 


10 




Esperanto 


1 




French 


25 




French-based Creole 


2 




Spanish 


17 




Portuguese 


4 




Italian 


5 




Welsh 


17 




German 


. 12 




Russian 


8 




Slovak 


4 




Polish 


1 




Serbo-Croatian 


3 




Greek 


10 




Kurdish 


1 




Farsi 


4 




Pushto 


2 




Urdu 


31 






nsa 

54 
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N 


% 


Punjabi 


28 




Hindi 


3 




Bengali/Bangla/Sylheti 


17 




Gujerati 


11 




Tamil 


7 




Sinhalese 


1 




Albanian 


1 




Arabic 


16 




Assyrian 


1 




Turkish 


9 




Wolof 


1 




Twi 


3 




Ga 


1 




Ashanti 


1 




Yoruba 


1 




Ibo 


4 




. Kiswahili 


3 




Somali 


1 




Chinese (Cantonese or Mandarin) 


18 




Japanese 


6 




Cambodian 


1 . 




Korean 


1 




Tagalog/Filipino 


2 




Fijian 


1 




Total 


296 




no information 


36 
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Previous basic skills tuition 


None 


639 


36 


In 1998 only 


419 


24 


In 1997 or earlier 


709 


40 


Total 


1767 


100 


no information 


148 





Forms of provision being attended in 1998/99 

(N.B. Percentages do not total to 100 because more than one category could be ticked) 


Primary-purpose literacy/ 
literacy & numeracy 


1357 


88 


Literacy/numeracy support 


273 


18 


Adult Training (Training for Work) 


100 


7 


Workplace training 


52 


3 


Training via Youth Credits 


22 


' 1 


Family literacy 


17 


1 


Total of responses 


1821 




Total of respondents 


1543 


100 


no information 


372 






46 Progress in Adult Literacy 



Modes of teaching being experienced in 1998/99 

(N.B. Percentages do not total to 100 because more than one category could be ticked) 


Croup tuition 


869 


68 


One-to-one provision in College/ 
Centre with tutor PLUS volunteer 
support 


282 


22 


Open learning 


240 


19 


One-to-one provision in College/ 
Centre with tutor 


238 


19 


Drop-in 


148 


12 


Total of responses 


1777 




Total of respondents 


1276 


100 


no information 


639 





Of the 206 intensive students for whom Profiles were returned, 107 (53 per 
cent) were men and 95 (47 per cent) were women (information was not provided 
for four people). Also, 108 (53 per cent) had never been in basic skills 
provision before, 55 (27 per cent) had attended in the previous year only, and 40 
(20 per cent) had attended in 1997 or earlier (there was no information on 18 
people). 



Thus in the full sample there were significantly more women than men (but there 
were slightly more men among the intensive students), a wide range of ages 
(up to 80+), and a balance of occupational statuses.* A great many of the 
students were monolingual English-speaking white people with few or no 
qualifications. The range of other languages spoken covered the globe; perhaps 
the most unusual item in the list was that 10 students gave Sign as their first 
language. Just over a third of the students in the full sample were new to basic 
skills provision, but many more had attended for at least one year previously, 
and some for many years - one fieldworker met a student who had been attending 
the same group for 1 7 years. The proportion of new students was much higher, and 
the proportion of long-term students was much lower, among the intensive 
students. 
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The large numbers for whom no information was given on forms of provision and 
modes of teaching were probably due to the structure of that section of the Profile, 
where these two topics were not distinguished. It may be suspected that most of 
those with no entry for a form of provision were actually attending primary-purpose 
provision: similarly that most of those with no entry for a mode of teaching were 
actually receiving group tuition. However, using the information that was available, 
it was possible to estimate that 223 (14 per cent) out of 1,543 students were 
attending more than one form of provision - but cf. the tutor questionnaire 
information in section 4.4. (The proportion receiving more than one mode of 
teaching was not calculated because of the large amount of missing information.) 



4.2 Hours of tuition 

The distribution of number of hours of literacy tuition attended between pre- and 
post-test is shown in Table 4.2. 



Table 4.2: Distribution of hours of literacy tuition between pre- and post-test 



Number of hours 


Number of students 


% 


more than 60 


164 


17 


51-60 


172 


18 ' 


41-50 


89 


9 


31-40 


154 


16 


21-30 


192 


20 


20 or less 


211 


21 


Total 


982 


100 



No information on hours of tuition was returned for 242 students for whom pre- 
and post-test reading data were received. The quite large groups with small 
numbers of hours were mainly students whose pre- and post-tests were not many 
weeks apart, or who were attending combined literacy and numeracy provision. 

4. 3 Comparisons with school-level attainment 

The tests used in the adult literacy study contained 22 tasks and 70 items 
(questions). Within these, three tasks (12 items) were drawn from those used in the 
1991 EEA Reading Literacy Study, Population A (9-year-olds) (Elley, 1992), and also 
used in the 1996 replication of that study with Year 4 pupils in England and Wales 
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(Brooks et al. 1996). Though these tasks were designed for 9-year-olds, their 
content was not so juvenile that they would have been inappropriate for adults. One 
of the three tasks (The Bird and the Elephant) provided the only narrative text used in 
this study. Another (Island) was a map-reading task of a sort familiar to many adult 
literacy students aiming for Wordpower accreditation. And Quicksand was of a 
familiar factual/expository genre. 

In the 1996 Year 4 study, the sample of children was nationally representative, and 
in particular covered the entire range of attainment. The IEA tasks and items used 
in the present study therefore provided a comparison between the performance of 
adults with poor literacy and Y4 pupils across the full ability range. The relevant 
data are shown in Table 4.3. 



Table 4.3: Comparative performance of adults and 9-year-olds on IEA Reading Literacy 
Study tasks 



Task & question 


IEA data 


Adult literacy data 




Pre-test 


Post-test 


N 


% correct 


N 


% correct 


N 


% correct 


Quicksand 


1 


917 


62 


594 


52 


295 


56 


2 


917 


71 


594 


51 


295 


57 


3 


917 


78 


594 


64 


295 


67 


Island 


1 


900 


83 


495 


69 


336 


71 


2 


900 


91 


495 


80 


336 


85 


3 


900 


62 


495 


61 


336 


70 


4 


900 


73 


495 


67 


336 


73 


The Bird and the Elephant 


1 


900 


46 


559 


72 


283 


76 


2 


900 


69 


559 


69 


283 


71 


3 


900 


36 


559 


54 


283 


64 


4 


900 


71 


559 


85 


283 


85 


5 


900 


86 


559 


93 


283 


95 
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Figure 4.1: Comparative performance of adults and 9-year-olds on IEA Reading Literacy 
Study Tasks' 




Because of various aspects of the design of the study (see Appendix B). the six sets of 
adult data in Table 4.3 (three tasks by two occasions of testing) were all provided by 
different groups of students. No student took all 12 items, and it was therefore not 
possible to produce comparative 'totals' for adults and children. Also, each of the 
three tasks was taken by a different group of adults within each test stage, and those 
who took The Bird and the Elephant at either stage had distinctly better literacy 
attainment than those who took Quicksand and Island. However, the percentages of 
adults and children getting the various items right did permit comparisons. 

In general, the percentages of adults getting items right on Island and especially on 
Quicksand were lower than the percentages of 9-year-olds who got them right; while 
the percentages of adults getting items right on The Bird and the Elephant were 
higher. Overall, therefore, the attainment of this sample of adults with poor literacy 
skills appeared to be similar to that of a full-range sample of 9-year-olds. In National 
Curriculum (NC) terms (cf. section E.2) this would put this sample of adults at NC 
level 2 on average. 
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4.4 Information from the tutor questionnaire 

4.4. 2 Students' literacy levels 

Tutors were asked how many of their students were at each level of the BSA's 
Communication Standards, from below Entry up to level 3. Just over 40 tutors did 
not answer this question; the results for the rest are shown in Table 4.4. 

Table 4.4: Numbers of students judged by tutors to be at different levels of BSA 
Communication Standards 



Level . 


Students 


Number of tutors with 
students at this level 


N 


% 


3 


68 


5% 


38 


2 


248 


18% 


100 


1 


504 


36% 


135 


Entry 


393 


28% 


127 


Below Entry 


156 


11% 


71 


Total 


1411 


100% 





Note: Percentages do not sum to 100 because of rounding errors. 



Thus tutors considered two fifths (39 per cent) of their students to have less 
than functional literacy (Entry level or below), and one in ten (11 per cent) to 
have very poor literacy (below Entry). Conversely, they considered three fifths 
(61 per cent) to be at or above the threshold of functional literacy (level 1 or above), 
and even one in twenty to be at or near the national average (level 3). These 
judgements seem somewhat generous when compared with the school-level 
comparisons reported in the previous section. Compared with the test data (see 
chapter 5) the students are also concentrated rather more in the middle of the 
scale. Both these findings are usual; human raters are reluctant to allocate many 
students to the extremes of a scale, and also exhibit what has been called ‘generosity 
error. 



4.4.2 Other literacy tuition received by tested group 

In their questionnaire, tutors were asked whether any students in their tested group 
were receiving other adult literacy teaching during the course of the week and, if so, 
how many. A total of 96 tutors (56 per cent of the 171 who answered this question) 
said their students were receiving other teaching, and between them reported 357 
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students (about a quarter of the relevant sample) as receiving other teaching. This 
was nearly double the proportion estimated from the Student Profile information - 
see section 4.1. 

In either case, only a minority of students attended more than one form of provision, 
and it therefore seems that most students received just a few hours' provision per 
week. Information from the tutor questionnaire, the fieldworkers’ impressions, and 
the data on hours of literacy tuition received by students between pre- and post-test, 
combined to suggest that the provision attended by the majority of students was 
non-intensive - a few hours a week over many months (and even years). However, 
in section 6.2 it will be shown that some students who attended more regularly 
made better progress in reading. 



4,5 Summary 

The sample of students in this study: 

• contained significantly more women than men (in the full sample, but among the 
intensive students there were slightly more men than women); 

• represented a range of ages and occupational statuses; 

• contained minorities of people for whom English was an additional language, 
and of ethnicities other than white; 

• were mostly poorly qualified; 

• had mostly attended basic skills provision before (in the full sample; among the 
intensive students just over half were new to adult literacy provision), and a 
minority were attending more than one form of provision in 1998/99; 

• mostly received more than 30 hours of tuition between pre- and post-test; 

• had reading levels quite similar, on average, to a national sample of 9-year-olds; 

• were judged by their tutors to range from very weak to about the national 
average, with a majority already above Entry level. 



Chapter 5 

How much progress did the students 
make? 

5 . 1 Reading - norm-referenced results 

The overall norm-referenced results for reading, on the IALS scale (mean 300. 
standard deviation 50), are shown in Table 5.1. 



Table 5.1 : Norm-referenced reading results 





Full sample 


Intensives 


N 




1224 


196 


Pre-test 


Mean 


214.3 


237.8 




(sd.) 


(76.8) 


(70.4) 




percentile 


19th 


30th 




Post-test 


Mean 


225.4 


246.3 




(s.d.) 


(72.1) 


(65.0) 




percentile 


22nd 


31st 




Gain 




11.1 


8.5 


Significance 




p<0.001 


p<0.01 


Effect size 




0.22 


0.17 



Key: N = sample size; mean = average score; s.d. = standard deviation 

Note: The effect sizes were calculated by dividing the gain by the s.d. of the reporting scale (50). 



The standard deviations were large, indicating a very wide dispersal of scores (see 
also the next section). The intensive students had distinctly higher average scores at 
both pre- and post-test than the full sample, and their smaller gain (shown 
particularly in the percentile data) was largely due to their having less room to show 
improvement. 



How much progress did the students make? 5 3 



Though the gains for both the full sample and the intensive students were 
statistically significant, this was partly because of the large sample sizes. Both 
gains were small, in absolute terms, and the effect sizes indicated what in an 
intervention study would be considered marginal effectiveness (the rule of thumb is 
that an effect size needs to be at least 0.25 to indicate educational significance). 
However, the Progress study at least was not an intervention study but what its 
name implies, a study of progress. Therefore the gain observed is what it was 
intended to discover. And that gain can be characterised as undramatic but 
worthwhile. 



5 . 2 Reading - criterion-referenced results 

Unsurprisingly (since the same data underlay both sets of results), the 
criterion-referenced results told the same story. The distributions of results on 
the IALS levels at pre- and post-test are shown in Table 5.2 for the full sample 
and in Table 5.3 for the intensive students. In both cases, the upper part of the 
table shows only the IALS levels, while the lower part subdivides IALS Level 1 
according to sub-levels 3, 2 and 1 of New Standards Entry Level. 

Information provided by the devisers of the New Standards showed that the 
Entry Level/Level 1 boundary in those standards had been aligned with the 
IALS Level 1 /Level 2 boundary. For the purposes of this study, it was further 
assumed that boundaries further up the two scales had also been aligned, as 
follows: 



New Standards boundaries 


IALS boundaries 


Level 2/Level 3 


Level 3/Level 4 


Level 1 /Level 2 


Level 2/Level 3 


Entry Level/Level 1 . 


Level 1 /Level 2 



Each of these levels was one IALS standard deviation (50 scale points) wide. IALS 
Level 1 had no subdivisions, but New Standards Entry Level was devised with 
three sub-levels. In order to estimate the proportions of students in this study falling 
into those sub-levels, the assumption about the ‘width’ of levels was also 
extrapolated downwards, that is, the top two of the three sub-levels within Entry 
Level were also treated as covering 50 scale points. Entry 1 then contained everyone 
below Entry 2. 
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Table 5.2: Criterion-referenced reading results, full sample 



Score 


IALS 


New Standards 


Pre-test 


Post-test 


range 


level 


level 


N 


% 


N 


% 


326-375 


4 


Level 3 


14 


1 


29 


2 


276-325 


3 


Level 2 


302 


25 


344 


28 


226-275 


2 


Level 1 


320 


26 


324 


27 


225 or below 


1 


Entry Level 


584 


48 


523 


43 


Total 






1220 


100 


1220 


100 


Within IALS Level 1/New Standards Entry Level: 


176-225 




Entry 3 


237 


19 


239 


20 


126-175 




Entry 2 


145 


12 


151 


12 


125 or below 




Entry 1 


202 


17 


133 


11 



Table 5.3: Criterion-referenced reading results, intensive students 



Score 


IALS 


New Standards 


Pre-test 


Post-test 


range 


level 


level 


N 


% 


N 


% 


326-375 


4 


Level 3 


3 


2 


5 


3 


276-325 


3 


Level 2 


70 


35 


74 


57 


226-275 


2 


Level 1 


56 


28 


58 


29 ' 


225 or below 


1 


Entry Level 


69 


35 


61 


31 


Total 






198 


100 


198 


100 


Within IALS Level 1/New Standards Entry Level: 


176-225 




Entry 3 


41 


21 


38 


19 


126-175 




Entry 2 


13 


7 


10 


5 


1 25 or below 




Entry 1 


15 


8 


13 


7 



The distributions again confirmed that the intensive students started and finished 
with rather better attainment than the full sample, and that students’ attainments 
were very diverse. Curiously, there were some students with pre-test scores above the 
national and international mean of 300, and therefore with reading skills 
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apparently above the average for the population. Table 5.2 also shows that just 
over half the full sample (52 per cent) were already above IALS Level 1/ 
New Standards Entry Level at pre-test, and Table 5.3 shows that this was 
particularly true of intensive students (65 per cent). This skew in the reading 
results towards the upper part of the range suggests that many students in the 
sample already had reasonable reading skills. However, 48 per cent of the full 
sample, and 3 5 per cent of intensive students, had less than functional literacy at 
pre-test, and sections 5.6-7 will suggest that many students had weaker writing 
than reading. 



At the lower end of the range, there were just over 200 students with pre-test 
scores of 125 or below (more than 3.5 standard deviations below the mean). To 
this extent, the tests had succeeded in ‘creating the lower rungs of the ladder’ (but see 
below). 

Another perspective on the criterion-referenced results for the full sample is given in 
Figure 5.1. Here, the distributions of pre- and post-test criterion-referenced scores 
are shown for each New Standards level and sub-level from Entry 1 up to Level 3, 
with the national distribution shown for comparison. The national distribution is a 
composite of the Prose and Document Literacy scales reported in Carey et al. 
(1997), but is more detailed than the data given by Carey et al. because it is based on 
the unpublished raw scores. 



Figure 5.1 shows that, as already discussed, quite a high proportion of students in 
this study already had pre-test scores above New Standards Entry Level. However, it 
also shows that the proportion of students in this study with scores within New 
Standards Entry Level was substantially higher than in the population, especially at 
Entry 1 . , 



Thus many of the students in this study were among those in greatest need. 
However, it has been estimated (Brooks et al., 2000, chapter 6) that the total 
number of adults in adult literacy provision is about 4 per cent of those 
estimated to have literacy skills within IALS Level 1/New Standards Entry Level 
and therefore thought to be in need. If the adults in this study are 
representative of all those in provision, then this suggests that the proportion of 
those in greatest need who are in provision is actually lower still - perhaps only 2 
per cent. 
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Figure 5.1 : Pre- and Post-test criterion-referenced reading results compared with national 
(IALS) distribution 




IALS r s “^ ,0-125 126-175 176-225, 226-275 276-325 326-375 376+ 



IALS level 1 2 3 4 5 



5.3 Comparisons with school-level results and tutors’ judgements 

Comparing the overall reading results with the school-level data presented in 
chapter 4 would not yield any further information, because the items taken from 
school-level tests were a subset of those on which the overall criterion-referenced 
results were based. 

However, it is possible, but difficult, to compare the attainment data in Table 5.2 
with the tutor report data in Table 4.4. The difficulty is that the scales used, 
though labelled similarly, were in fact different. When the tutors were asked to give 
their estimates, the question was put in terms of the Basic Skills Agency’s 
Communication Standards, which were then the only ones current; but by the time 
the criterion-referencing reported in this chapter was carried out, the new Basic 
Skills Standards had become available in draft (Qualifications and Curriculum 
Authority, 1999). and these were used for the criterion-referencing. The new 
Standards were aligned with the IALS levels and, as pointed out by Carey et al 
(1997. p.17). the level boundaries in IALS were slightly but significantly higher 
than those in the BSA Standards. 
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That said, the attainment data here and the tutor report data in Chapter 4 
provided rather different ‘averages’ (the tutor judgements being somewhat 
higher), but similar pictures of the overall range of attainment: a heterogeneous 
group of adults with skills ranging from extremely poor to about (or even just 
above) the national average, the majority having at least functional literacy. 

It might therefore be wondered why so many people with apparently adequate 
skills were in adult literacy provision. Sketchy answers were available from the 
fieldworkers’ impressions. At least a few of the tested groups of students 
were attending courses with a specific focus on ICT, and pre-test attainment in 
those groups seemed particularly high; perhaps some of these students were 
attending more to acquire ICT skills than because their literacy needing 
improvement. Many adults with level 1 literacy appeared to go on attending in order 
to maintain their skills, perhaps particularly their (probably weaker) writing: some 
might have felt that without continuing support their hard-won skills might 
deteriorate again. 



5,4 Movement between levels 

Tables 5.2 and 5.3 both showed a modest amount of progress, on average. This is 
confirmed by the data shown in Tables 5.4-6. All the results in those Tables suggest 
considerable ‘turbulence’ within the samples - but this is usual in studies of this 
kind. That is, every instrument for measuring reading attainment has some degree 
of unreliability - if a group of students were re-tested the next day on the same 
instrument a proportion of them would score differently, some because they 
remember or work out an answer they got wrong the day before, others because 
they forget or mistakenly change one they got right, etc. Moreover, in this study no 
student saw the same questions at pre- and post-test, so that two sources of 
unreliability come into play. However, here the correlation between the pre- 
and post-test scores for the full sample was 0.75, which is reasonable given the 
small number of items which any one student took at either stage (maximum of 24 
or 26). 

Table 5.4 shows the numbers of students whose score did or did not change between 
pre- and post-test by at least a quarter of a standard deviation (±12.5 points) on the 
IALS scale - a quarter of a standard deviation is considered to be the minimum 
change that is educationally significant. On this criterion, distinctly more students 
improved significantly than fell behind. 
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Table 5.4: Numbers of students who moved ±12.5 IALS scale points or stayed within that 
range 





Full sample 


Intensives 


N 


% 


N 


% 


Moved up at least 12.5 points 


606 


50 


93 


47 


Stayed within ±12.5 points 


247 


20 


41 


21 


Moved down at least 1 2.5 points 


357 


30 


62 


32 


Total 


1210 


100 


196 


100 



Tables 5.5 and 5.6 show (for the full sample and intensive students respectively) the 
numbers of students who did or did not move between levels of the New Standards. The 
numbers in bold on the diagonals are students who remained at the same level: those to 
the right of the diagonal are students who fell at least one level: and those to the left rose 
at least one level. Again, considerably more students improved than fell behind. 

Table 5.5: Numbers of students who moved between levels of the New Standards or 



stayed at the same level, full sample 





Pre-test level 


Entry 1 


Entry 2 


Entry 3 


Level 1 


Level 2 


Level 3 


Total 


Post-test level 
















Level 3 


1 


0 


2 


7 


19 


0 


29 


Level 2 


6 


5 


41 


114 


166 


12 


344 


Level 1 


12 


25 


76 


121 


89 


1 


324 


Entry 3 


27 


57 


78 


55 


22 


0 


239 


Entry 2 


55 


40 


34 


20 


2 


0 


151 


Entry 1 


101 


18 


6 


3 


4 


1 


133 


Total 


202 


145 


237 


320 


302 


14 


1220 



In the full sample, in terms of New Standards levels (including sub-levels of Entry Level): 

• 447 students (3 7%) moved up at least one level 

• of these, 126 (10%) moved up 2 or more levels 

• 506 (41%) stayed on the same level 

• 267 (22%) dropped at least one level 

• of these, 59 (5%) dropped 2 or more levels. 
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Table 5.6: Numbers of students who moved between levels of the New Standards or 



stayed the same, intensive students 





Pre-test level 


Entry 1 


Entry 2 


Entry 3 


Level 1 


Level 2 


Level 3 


Total 


Post-test level 
















Level 3 


0 


0 


0 


0 


.5 


0 


5 


Level 2 


. 0 


1 


5 


23 


42 


3 


74 


Level 1 


1 


1 


15 


25 


16 


0 


58 


Entry 3 


1 


9 


14 


7 


7 


0 


38 


Entry 2 


3 


2 


4 


1 


0 


0 


10 


Entry 1 


10 


0 


3 


0 


0 


0 


13 


Total 


6 


10 


45 


101 


36 


3 


198 



Of the intensive students, in terms of New Standards levels (including sub-levels of 
Entry Level): 

• 64 (32%) moved up at least one level 

• of these, 9 (5%) moved up 2 or more levels 

• 93 (47%) stayed on the same level 

• 41 (2 1%) dropped at least one level 

• of these, 10 (5%) dropped 2 or more levels. 



5 . 5 What could students at each reading level do? 

Tabulating the percentages of students who were at each level on the 
IALS/New Standards scale gives no sense of what tasks students at different 
levels could or could not achieve in terms of reading. In IALS, student 
performance at each level was characterised in terms of what students at that 
level had an 80 per cent probability of being able to do. Therefore listing items 
which students at a level had an 80 per cent probability of being able to do 
gave a general impression of the kinds of literacy skill which people at that 
level possessed. Inspection of the facility values for all the items in this study 
provided the lists in Table 5.7; the lists are cumulative from level to level. 
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Table 5.7: What students at each level had an 80 per cent probability of being able to do 

Students in this study who had scores in the IALS Level 1/New Standards Entry 
Level range (225 or less) had an 80 per cent probability of being able to: 

• ring dates on a calendar 

• locate the amount of milk needed for a Custard recipe 

• locate information in a note from a neighbour. 

Students in this study who had scores in the IALS Level 2/New Standards Level 1 
range (226-275) also had an 80 per cent probability of being able to: 

• locate phone numbers 

• locate the amount of sugar needed for a Custard recipe 

• retrieve information about quicksand 

• retrieve information about seedsticks 

• retrieve information from a story about The Bird and the Elephant 

• retrieve simple information from a notice about a meeting 

• locate information in yellow pages 

• answer questions about a simple map 

• use a conversion table (gallons and litres) 

• locate information in a charity advertisement (IALS Level 1, Prose) 

• answer a simple question about a union ballot notice (IALS Level 1, Document) 

• retrieve simple information from a medicine label (IALS Level 1, Prose) 

• say why a Scrambled Eggs recipe calls for sugar (IALS Level 1, Prose) 

• locate other information in a Scrambled Eggs recipe 

• transcribe the distance from home to work for an employment application (IALS Level 1 , 
Document) 

• transcribe other simple information for an employment application (IALS Level 2, 
Document) 

• underline information in a newspaper article (IALS Level 1, Prose) 

• retrieve simple information from a newspaper article (IALS Level 2, Prose) 

• fill in the price and number of tickets required on a form for theatre tickets (IALS Level 1 , 
Document) 




How much progress did the students make ? 6 1 



• fill in the date and time on a form for theatre tickets (IALS Level 2, Document) 

• answer a simple question about a Nuclear Waste chart (IALS Level 1, Document) 

• answer other simple questions about charts (IALS Level 2, Document) 

• retrieve single-point information about fighting fires (IALS Level 1, Prose). 

Students in this study who had scores in the IALS Level 3/New Standards Level 
2 range (276-325) also had an 80 per cent probability of being able to: 

• locate which of three dates was relevant to a question about a union ballot notice 

• choose the best title for a piece about quicksand 

• make a simple inference from information about seedsticks 

• make simple inferences from a story about The Bird and the Elephant 

• locate the date of the lowest point in a chart (IALS Level 2, Document) 

• make a simple inference for an employment application form (IALS Level 2, Document) 

• make a difficult inference about a new law on fighting fires (IALS Level 2, Prose) 

• retrieve two pieces of information about fighting fires (IALS Level 3, Prose) 

• relate two pieces of information from a chart (IALS Level 3, Document) 

• make an inference about information in a Nuclear Waste chart (IALS Level 3, Document). 

Students in this study who had scores in the IALS Level 3/New Standards Level 
2 range (276-325) had less than an 80 per cent probability of being able to: 

• state when a recipe said custard should be stirred 

• combine and transform two pieces of information for an employment application form 
(IALS Level 3, Document) 

• re-express one of seven points in a chart (IALS Level 3, Document) 

• transcribe three pieces of information from a medicine label (IALS Level 3, Prose). 



The first part of Table 5.7 shows that even students approaching the top of the New 
Standards Entry Level could generally cope with only a few items, these being very 
simple information-retrieval tasks. Students at Entry Levels 1 and 2 generally could 
not cope even with these items - and students at these levels were 29 per cent of the 
full sample at pre-test and 23 per cent at post-test. It would seem fair to say that 
none of the students below IALS Level 2 /New Standards Level 1 had yet achieved 
functional literacy - and students at this level comprised 48 per cent of the full 
sample at pre-test and 43 per cent at post-test. 
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The very short list of items which students at IALS Level 1/New Standards 
Entry Level could generally manage suggests that, despite the success in 
achieving finer statistical discrimination at the lower end of the scale, 
the ‘lower rungs of the ladder were still not numerous enough for the students with 
the lowest levels of literacy. The implication is that, for the weakest students, 
there need to be ‘very small steps’ on which they can demonstrate progress. 

Students at IALS Level 2 /New Standards Level 1 could generally cope with a much 
wider range of information-retrieval items, including many where they had to write 
their own answers (those labelled as IALS items in Table 5.7), but not yet with items 
requiring inference or the relating of separate pieces of information - even the 
question about why a scrambled eggs recipe called for sugar required only the 
location of the relevant sentence in the recipe. Students at this level could be said to 
have largely achieved functional literacy - but not yet to be able to cope with more 
demanding literacy tasks. The ‘turbulence' in the data mentioned above suggests, 
however, that many students who achieved IALS Level 2/New Standards Level 1 
were not yet fully secure in their attainment of that level. 

Students at IALS Level 3 /New Standards Level 2 could generally cope with items 
requiring them to make inferences or relate separate pieces of information or 
distinguish relevant from distracting information - in other words, they had 
quite well developed reading skills, Since these students had scores only just below 
the national average their skill level was perhaps unsurprising - but their presence 
in, or need to attend, basic skills provision might be thought surprising. They may 
have been people who considered their existing near-average skills not yet good 
enough for their aspirations in life or employment, particularly, perhaps, their 
writing skills. 

No attempt is made to characterise the performance of the very small number of 
students who scored at IALS Level 4/New Standards Level 3. 

The last part of Table 5.7 shows the few tasks that even the highest-scoring students 
in the sample were generally not yet able to do - these were tasks requiring complex 
inferences or the interpretation of difficult text. 

In the IALS study, adults who achieved scores at a particular level were by 
definition those who could generally manage the items whose facility values 
were also at that level. It might therefore be thought that the same would apply to 
the students in this study, where values on the IALS scale for non-IALS items 
were derived from those for IALS items via the performance of students talcing 
samples of both sets of items. However, Table 5.7 shows that this was not necessarily 
the case. In the Table, tasks derived from IALS are labelled with their level and as 
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Prose or Document (this distinction is explained in Appendix B). The labelling by 
level reveals a curious aspect of the results: students in this study at the various IALS 
levels as shown by their scores were not necessarily generally able to cope with tasks 
which the IALS study itself showed could be managed by people at those levels 
within a nationally representative full-range sample: 

• students in this study with IALS Level 1 scores could not generally manage any of 
the IALS Level 1 tasks; 

• students in this study with IALS Level 2 scores could generally manage all the 
IALS Level 1 tasks, but not all the Level 2 tasks; 

• students in this study with IALS Level 3 scores could generally manage all the 
IALS Level 2 tasks, but not all the Level 3 tasks. 

This is an odd result. Students in this study exhibited different literacy behaviour 
than adults in the same part (mainly the lower half) of the national distribution: 
they achieved scores at a higher IALS level than their performance on the IALS 
items alone would have earned. 

Part of the explanation may lie in the turbulence in the results already 
mentioned. 

A possible, more general explanation of this pattern might relate to 
differential performance on the Prose and Document items. The difference 
in performance on the two classes of IALS items is not visible in the list above 
but only in the detailed item statistics (see Appendix E). The proportions of 
students in this study who produced the correct answers to IALS items were 
closer to the national proportions on Document than on Prose items. They 
could cope better with non-continuous texts (such as application forms for 
employment and for theatre tickets, and graphic • information in charts) than 
with continuous texts (such as a dense piece about fighting fires, a 
newspaper article, etc.). Thus basic skills students seem to have learnt to cope 
better with the sorts of literacy skills that are also life skills, but not at all well 
with stretches of prose - which may require more sustained attention and 
more focused skills. It may also be that the students in this study are 
unrepresentative of adults in the lower half of the national distribution of literacy 
attainment in this disjunction of reading skills. 

5.6 Results for writing 

As described in Appendix F, the students' writing samples were analysed in terms 
of length (number of words), numbers of errors in various categories, and quality 
of handwriting. Somewhat surprisingly, no significant changes between pre- 
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and post-test were found for the error categories (Grammar, Style, Spelling 
and Other Orthographic Conventions), not even for the total number of errors 
per 100 words. Thus students on average did not improve their writing in terms 
of a reduction in errors or an increase in lexical or syntactic complexity. Results 
are therefore given only for length of script and quality of handwriting, and only 
for the full sample - see Tables 5.8 and 5.9. 



The very small average increases, of two words in length of script and a small 
fraction of a point in quality of handwriting, were nevertheless statistically 
significant for the full sample. For the intensive students, there was a similar 
statistically significant increase in length of script, but not in quality of handwriting. 
Some students were evidently quite prolific writers, but the majority (56 per cent) 
produced 20 words or less at pre-test. Compared to the gains in reading, the 
improvements in writing seemed even more undramatic. The improvement in 
handwriting for the full sample was so marginal that it seems educationally 
insignificant (despite being statistically significant), but the increases in length of 
script seem worthwhile. 



Table 5.8: Writing results: length of script, full sample 





Pre-test 


Post-test 




Number of words written 


N 


% 


N 


% 




>80 


20 


2 


20 


2 




71-80 


11 


1 


14 


1 




61-70 


23 


2 


21 


2 




51-60 


30 


3 


36 


4 




41-50 


50 


5 


73 


8 




31-40 


90 


10 


115 


12 




21-30 


187 


20 


198 


21 




11-20 


269 


29 


251 


27 




1-10 


205 


22 


164 


18 




0 


51 


5 


45 


5 




Total 


937 


100 


937 


100 




Mean 


23.0 




25.1 




p<0.003 


(s.d.) 


(19.4) 




(19.1) 
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Table 5.9: Writing results: quality of handwriting, full sample 





Pre-test 


Post-test 




Quality of handwriting 


N 


% 


N 


% 




3 


563 


69 


590 


72 




2 


192 


23 


187 


23 




1 


66 


8 


44 


5 




Total 


821 


100 


821 


100 




Mean 


2.61 




2.67 




p<0.02 1 


(s.d.) 


(0.63) 




(0.58) 







Note: The 1 16 students 'missing' from Table 5.9 relative to Table 5.8 were excluded from this analysis because their 
scripts were too short to permit the quality of handwriting to be reliably assessed. 



5.7 Relationship between reading and writing results 

Table 5.2 showed that many students in the full sample already had quite 
reasonable reading skills at pre-test. Table 5.8, on the other hand, showed that most 
students at pre-test wrote very brief scripts. This suggests that many students in 
adult literacy provision have greater problems with writing than with reading, and 
that there is a need for investigation of the reasons for the apparent discrepancy, and 
of the implications for teaching. 



5.8 Summary 

• Finer statistical discrimination at the lower end of the reading assessment scale 
was achieved. However, there were still too few reading items which the very 
weakest students could manage. 

• Both the full sample and the intensive students achieved a small but statistically 
significant improvement in reading. 

• The average improvement in reading for the full sample represented a move from 
the 19th to the 22 nd percentile of the national distribution. 

• The average improvement in reading for the intensive students represented a 
move from the 30th to the 31st percentile of the national distribution. 
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• In reading, 48 per cent of the fLill sample were not reading at a functional level at 
pre-test, and within these 29 per cent had very low pre-test scores. However, just 
over half were already reading at functional level, and a few students were already 
above the national average. Even so, these findings suggest that an even smaller 
proportion of adults in greatest need of help with literacy are being reached than 
previously thought. 

• The ‘turbulence’ seen in the results is an inherent aspect of all cognitive testing, but 
suggests that some students were not securely above the threshold of functional level. 

• In the reading tests, students within Entry Levels 1 and 2 of the new Basic Skills 
Standards (29 per cent of the full sample) could not generally manage any of the test 
items. 

• Students within Entry Level 3 of the new Basic Skills Standards could generally 
manage only a few very simple information-retrieval items. 

• Students within Level 1 of the new Basic Skills Standards could generally manage all 
the information-retrieval reading items, but none requiring higher-order skills. 

• Students within Level 2 of the new Basic Skills Standards could generally manage all 
the information-retrieval reading items and most of those requiring higher-order 
skills. 

• Many students seemed to have developed coping strategies for reading non- 
continuous ‘official-type texts, but could cope less well with continuous texts such as 
newspaper articles. They may not be representative of adults in the lower half of the 
national distribution of reading skills in this respect. 

• Both the full sample and the intensive students achieved a very small but statistically 
significant improvement in writing in terms of length of script , and the full sample 
also in quality of handwriting . However, these were the only statistically significant 
findings for writing; no significant improvements were found in terms of reduction of 
envrs or increase in complexity . 

• Many students in the sample seemed to have distinctly poorer writing than reading 
skills. This phenomenon needs to be investigated further, along with its implications 
for teaching. 

• Several of these findings lead to the conclusion that many students’ literacy 
attainments are uneven, between reading and writing, between types of reading 
material, and from one occasion to another. 

• Some students seemed to attend as much to maintain their skills as to improve them. 

• The main finding was that students made average gains in both reading and 
writing that were undramatic but worthwhile . 



Chapter 6 



What factors were associated with 
progress? 

6. 1 Few factors were discovered 

In the search for factors which might be associated with different amounts of 
progress, the reading and writing results were related to the background variables 
from the Student Profiles, plus number of hours of tuition attended between pre- 
and post-test, and several items from the tutor questionnaire. The analyses against 
background variables and hours of tuition were done both for the full sample and 
for the subsample of intensive students, those against tutor questionnaire data just 
for the full sample. 

Few factors related to differences in progress were discovered. The results for just the 
intensive students showed no statistically significant amounts of progress for 
subgroups either for reading or for writing. This was almost certainly due to the 
small numbers in each subcategory. However, no conclusions can be drawn from 
statistically non-significant differences, and results for the intensive students are 
therefore not reported here. (However, one possible factor relating to hours of tuition 
on intensive courses did emerge from the analyses for the full sample - see section 
6.4). 



6.2 Reading results against background variables, full sample 

The reading results for the full sample against the major background variables 
are shown in Table 6.1. Results for individual languages other than English 
are not shown because the numbers in each subcategory were very small. 
Under qualifications, all subcategories other than ‘none’ have been collapsed 
into the category labelled ‘some’ because of small numbers. Results for 
varieties of provision attended and modes of teaching experienced in 1998/99 
are also not shown, because some students , were attending more than one 
form of provision, and some undoubtedly were experiencing more than one 
mode of teaching, even though the proportion could not be calculated (see 
chapter 4). 
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Table 6.1 : Norm-referenced reading results against background variables, full sample 





N 


Pre-test 


Post-test 


Gain 


P 


Mean 


(s.d.) 


Mean 


(s.d.) 


Gender 


Men 


495 


206.9 


(78.6) 


218.2 


(70.2) 


11.3 


0.000 


Women 


603 


223.2 


(75.3) 


232.7 


(73.7) 


9.5 


0.000 


Total 


1098 














Age 


50+ 


200 


205.9 


(77.0) 


216.6 


1 (72.4) 


10.7 


0.016 


40-49 


213 


215.3 


(78.1) 


225.4 


(75.9) 


10.1 


0.013 


30-39 


290 


227.2 


(74.7) 


235.8 


(70.1) 


8.6 


0.015 


20-29 


167 


210.3 


(79.2) 


222.9 


(73.1) 


12.6 


0.002 


under 20 


136 


222.8 


(71.2) 


233.0 


(68.7) 


10.2 


0.027 


Total 


1006 














First language 


English 


937 


215.4 


(78.6) 


226.1 


(72.7) 


10.7 


0.000 


Other 


130 


210.7 


(70.2) 


225.5 


(72.7) 


14.8 


0.015 


Total 


1067 














Occupational status 


Employed 


284 


222.4 


(77.6) 


232.7 


(72.7) 


10.3 


0.005 


Unemployed 


284 


208.3 


(79.2) 


216.8 


(73.4) 


8.5 


0.021 


F/t education 


114 


221.7 


(69.8) 


230.9 


(69.4) 


9.2 


0.043 


Sick/disabled 


132 


217.1 


(76.8) 


230.6 


(66.1) 


13.5 


0.005 


Looking after home/family 


164 


238.3 


(66.7) 


244.8 


(67.2) 


6.5 


ns 


Other 


106 


190.6 


(84.5) 


207.6 


(83.8) 


17.0 


0.002 


Total 


1084 














Highest qualification 


None 


537 


207.9 


(78.5) 


220.9 


(73.7) 


13.0 


0.000 


Some 


581 


222.6 


(74.7) 


231.1 


(70.7) 


8.5 


0.000 


Total 




1118 












Previous basic skills tuition 


None 


350 


238.8 


(65.9) 


248.7 


(68.9) 


9.9 


0.004 


In 1998 only 


248 


218.7 


(77.2) 


225.4 


(69.5) 


6.7 


0.045 


Earlier 


466 


193.2 


(80.7) 


206.0 


(73.2) 


12.8 


0.000 


Total 


1064 















Key: N = sample size; mean = average score; s.d. = standard deviation; p = probability; ns = statistically non-significant 
Note: The differing total Ns for subcategories are due to missing data. 
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Almost all the subgroup results for the full sample were statistically significant. 
Thus progress in reading was made by both men and women students, those of 
different age groups, those with English as first or an additional language, those in 
different occupational statuses (except those looking after home and family), those 
with some formal qualifications and those with none, and those with differing 
amounts of previous basic skills tuition. In brief, almost all subgroups benefited, on 
average. 

In the case of those looking after home and family, the likeliest explanation of the 
lack of statistical significance was that this group already had a relatively high 
average score at pre-test (238.3), and the statistics of the instruments used meant 
that there was little room for their average score to increase. 



6.3 Writing results against background variables 

Pre- and post-test writing samples were returned for fewer students than reading 
tests. Also, some student's scripts were blank, or had notes attached saying they had 
been scribed by the tutor. Hence the substantially lower overall numbers of students 
in the analyses of writing than of reading. 

All the analytic categories described in Appendix F (length of script; handwriting; 
errors per 100 words overall and for the subcategories Grammar, Style, Spelling and 
Other Orthographic Conventions) were investigated in relation to background 
variables. Very few statistically significant differences were found. Most of those that 
were found related to length of script, and the very few significant results for other 
variables formed no pattern. The only category of writing analysis reported here is 
therefore length of script, and the results against the major background variables for 
this aspect of writing for the full sample are shown in Table 6.2. Under qualifi- 
cations, all subcategories other than ‘none' have again been collapsed into the 
category labelled ‘some’ because of small numbers. 
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Table 6.2: Writing (length of script) results against background variables, full sample 





N 


Pre-test 


Post-test 


Cain 


P 


Mean 


(s-d.) 


Mean 


(s-d.) 


Gender 


Men 


364 


22.1 


(18.9) 


24.5 


(18.0) 


2.4 


0.019 


Women 


433 


26.4 


(19.3) 


27.8 


(19.2) 


1.4 


ns 


Total 


797 














Age 


50+ 


148 


24.6 


(20.0) 


22.9 


(16.4) 


-1.7 


ns 


40-49 


162 


24.4 


(20.2) 


27.9 


(20.6) 


3.5 


0.040 


30-39 


205 


23.6 


(17.0) 


27.3 


(16.8) 


3.7 


0.007 


20-29 


117 


24.3 


(18.8) 


29.4 


(22.2) 


5.1 


0.023 


under 20 


113 


25.2 


(20.5) 


24.8 


(17.1) 


-0.4 


ns 


Total 


745 














First language 


English 


696 


24.8 


(19.6) 


26.5 


(19.0) 


1.7 


0.030 


Other 


89 


21.7 


08.0) 


25.3 


(18.2) 


3.6 


ns 


Total 


785 














Occupational status 


Employed 


203 


22.1 


(19.0) 


26.9 


(19.2) 


4.8 


0.002 


Unemployed 


193 


23.9 


(19.0) 


26.3 


(18.8) 


2.4 


ns 


F/t education 


94 


28.5 


(21.5) 


23.7 


(16.8) 


-4.8 


0.025 


Sick/disabled 


96 


22.9 


(20.8) 


26.2 


(20.6) 


3.3 


ns 


Looking after home/family 


117 


25.0 


(16.0) 


29.2 


(19.0) 


4.2 


0.028 


Other 


82 


26.7 


(21.2) 


24.9 


(17.7) 


-1.8 


ns 


Total 


785 














Qualifications 


None 


380 


23.5 


(19.0) 


25.5 


(18.7) 


2.0 


ns 


Some 


427 


25.4 


(19.6) 


27.1 


(18.9) 


1.7 


ns 


Total 


807 














Previous basic skills tuition 


None 


247 


26.0 


(19.3) 


28.7 


(18.7) 


2.7 


ns 


In 1998 only 


187 


26.4 


(22.2) 


27.5 


(19.8) 


1.3 


ns 


Earlier 


338 


22.8 


(18.1) 


24.1 


(17.8) 


1.3 


ns 


Total 


772 















Key: N = sample size; mean = average score; s.d. = standard deviation; p = probability; ns = statistically non-significant 
Note: The differing totals for subcategories are due to missing data. 
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The overall increase in length of script was so small (see section 5.6) that it is not 
surprising that few subgroups showed statistically significant differences between 
pre- and post-test. The significant differences that were found fell into no discernible 
pattern. 

6.4 Reading and writing results by hours of tuition 

The results against hours of literacy tuition attended between pre- and post-test are 
shown in Table 6.3 for reading and Table 6.4 for writing. 



Table 6.3: Norm-referenced reading results against hours of tuition, full sample 



Hours of tuition 


N 


Pre-test 


Post-test 


Cain 


P 


Mean 


(s.d.) 


Mean 


(s.d.) 


more than 60 


164 


197.8 


(79.4) 


217.1 


(74.2) 


19.3 


0.000 


51-60 


172 


195.9 


(71.8) 


219:7 


(68.0) 


23.8 


0.000 


41-50 


89 


228.6 


(73.0) 


236.4 


(62.6) 


7.8 


ns 


31-40 


154 


214.6 


(70.1) 


225.5 


(63.2) 


10.9 


0.015 


21-30 


192 


206.1 


(83.9) 


227.9 


(76.8) 


21.8 


0.000 


20 or less 


211 


204.9 


(79.6) 


222.6 


(75.1) 


17.7 


0.000 


Total 


982 















Key: N = sample size; mean = average score; s.d. = standard deviation; p = probability; ns = statistically non-significant 



Table 6.4: Writing results (length of script) against hours of tuition, full sample 



Hours of tuition 


N 


Pre-test 


Post-test 


Gain 


P 


Mean 


(s.d.) 


Mean 


(sd.) 


more than 60 


114 


30.6 


(23.6) 


28.7 


(18.6) 


-1.9 


ns 


51-60 


49 


22.3 


(17.7) 


21.5 


(14.8) 


-0.8 


ns 


41-50 


56 


28.5 


(21.0) 


23.7 


(15.6) 


-4.8 


ns 


31-40 


92 


23.4 


(16.6) 


27.1 


(18.5) 


3.7 


ns 


21-30 


121 


19.3 


(15.5) 


22.3 


(14.9) 


4.0 


ns 


20 or less 


128 


21.6 


(17.5) 


24.0 


(17.1) 


2.4 


ns 


Total 


560 















Key: N = sample size; mean = average score; s.d. = standard deviation; p = probability; ns = statistically non-significant 
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The numbers of students in the subgroups for writing were evidently too small to 
allow statistically significant differences to emerge. For reading all the subgroups 
except one made significant progress, and the lack of statistical significance in that 
case was again almost certainly due to the small number involved. Within the 
reading data the largest average improvement was shown by the 51-60 hours of 
tuition group. Given that the maximum number of teaching weeks between pre- 
and post-test was about 20, students in this group must have attended very 
regularly. This group's average gain was 23.8 scale points. This was the largest 
subgroup gain in the study and, at nearly half a standard deviation, probably the 
most educationally significant. Also, since the students on intensive courses for 
definition attended regularly, the finding on the benefit of regular attendance 
suggests that intensive courses could be particularly effective. 



6 . 5 Reading gains against tutor questionnaire variables 

No analyses were done of writing gains against tutor questionnaire information 
because the only writing gains that were significant overall were length of script and 
quality of handwriting. There seemed little interest or point in relating tutors' 
responses to either of these. 

Because of the circumstances in which the tutor questionnaires were distributed, no 
system could be set up for relating individual tutors’ responses to the scores of 
students in their tuition groups. So all the analyses in this section were based on 
averaging the responses of tutors within providers, and then relating those averaged 
responses to average reading gains of students within relevant providers. 

It proved possible to relate tutor responses to reading gain data for 1051 students - 
this was 86 per cent of all those for whom reading gain data were available. 

Many items from the tutor questionnaire could not be meaningfully related to 
reading gains, mainly because of imbalances in the questionnaire responses. For 
instance, so few tutors did not have a qualification for teaching basic skills that 
analysing reading gains against whether tutors did or did not have such a 
qualification would inevitably have produced a non-significant result. In some other 
cases, reading gains could not be related to tutor questionnaire variables because of 
overlaps in the response categories. 

A total of 1 7 items on the tutor questionnaire were therefore identified which, in 
the project director’s judgement, would be central to the interpretation of the 
overall gains if they showed significant effects, and reading gains were 
statistically related to these items. Only two of these produced statistically 
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significant findings. Because of this, and to avoid a ‘fishing expedition’ for isolated 
results which might attain statistical significance by chance, the search for 
relationships between reading gains and tutor questionnaire variables was 
terminated at that point. Table 6.5 lists the factors which produced non-significant 
findings. In each case the question number in the tutor questionnaire (Appendix D) 
is given for reference. 

Table 6.5: Items from tutor questionnaire which yielded non-significant results against 
reading gains 

• tutors having a degree or not, basic skills qualification or not (q.T.2) 

• tutors' years of teaching experience (q. 1 .4) 

• frequency of students' access to IT (q.3.8) 

• percentage of teaching time in which tutors used textbooks, etc., or their own materials 
(qs.3.10 & 3.12) 

• the amount of management support for tutors (q.5. 1 ) 

• the frequency of formal and informal staff meetings (qs.5.2 & 5.3) 

• whether tutors followed a defined curriculum or a scheme of work (qs.6.1 & 6.3) 

• the basis of the curriculum or scheme of work (qs.6.2 & 6.4) 

• aspects of tutors' planning of their teaching and assessment (q.6.5) 

• aspects of literacy included in scheme of work (q.7. 1 ) 

• teaching individuals versus small groups (q.7.2) 

• perceived importance of factors in students' progress (q.7. 4). 



Care is needed in the interpretation of the evidence summarised in Table 6.5. 
Factors for which no statistically significant association was found may still be 
important - it may just have been that the statistics did not pick this up on this 
occasion. 

The only two items which did produce statistically significant findings were whether 
tutors had volunteers and/or paid assistants (q.4.1) and whether all tutors in an 
area had qualified teacher status (QTS) (q.1.2). The classroom assistance item 
required only a Yes/No response from each tutor, but the analysis nevertheless had 
to be in terms of three categories: All tutors, Some tutors, No tutors. This is because 
tutors' responses had to be averaged within providers (because of the impossibility, 
already mentioned, of relating individual tutors' responses to their students' scores). 
The Some tutors category therefore represented providers where more than one 
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tutor responded and their answers were not all either positive or negative. The 
results on this variable are shown in Table 6.6. 



Table 6.6: Reading gains by proportion of tutors with classroom support 



Proportion of 
tutors with support 


N of 
students 


Pre-test 


Post-test 


Cain 


Average (s.d.) 


Average (s.d.) 


All tutors 


415 


214.5 


(77.3) 


230.3 


(72.1) 


15.8 


Some tutors 


486 


208.6 


(77.6) 


220.0 


(70.7) 


11.4 


No tutors 


150 


237.0 


(63.0) 


237.3 


(63.4) 


0.3 


Total 


1051 













The result was statistically significant, p=0.013. Students in providers where no 
tutors had support made virtually no average gain. Though it may seem obvious 
that tutors having classroom help would enable students to make better progress, 
this finding gives that belief statistical backing. 

For the OTS item it was found that all providers had at least some tutors with OTS, 
so the analysis was in terms of providers where all tutors had QTS versus those 
where not all tutors had OTS. The results on this variable are shown in Table 6.7. 



Table 6.7: Reading gains by whether all tutors had QTS or not 



Proportion of 
tutors with QTS 


N of 

students 


Pre-test 


Post-test 


Gain 


Average (s.d.) 


Average (s.d.) 


All tutors 


512 


214.1 


(73.8) 


231.2 


(66.7) 


17.1 


Some tutors 


530 


214.8 


(78.4) 


224.2 


(71.9) 


9.4 


Total 


1042 













The result was statistically significant, p=0.02. Students in providers where all tutors 
had OTS made more gain on average than those elsewhere. This would appear to 
support the case for the professionalisation of the adult literacy teaching force. 

6.6 Insights from interviews with basic skills co-ordinators 

When the main data analyses were complete, the reading and writing results were 
further analysed according to providers, in order to see whether students in some 
providers' areas made statistically significant progress. In most cases, the number of 
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students tested in an area was too small to yield a significant result, and this must be 
borne in mind in interpreting the opinions reported in this section. However, in no case 
did the students in a particular area make a statistically significant loss, and in seven 
cases students in particular areas did make statistically significant progress. The Basic 
Skills Co-ordinators in those seven providers were contacted and interviewed by 
telephone - but were not told the basis of selection. This stage of the project was an 
attempt to glean qualitative insights into what made the provision for which these co- 
ordinators were responsible, particularly effective. 

Most of the picture was as would be expected. Most provision was based on the teaching 
of groups of students (though there was much individualisation of teaching within 
group provision). All the providers catered for special groups of learners, worked in 
partnership with a range of organisations, derived the bulk of their funding from the 
FEFC, and set City and Guilds 9281 as the minimum qualification for tutors. Most 
tutors were part-time - in fact, in one of these seven providers all the adult literacy staff 
were part-time. Students’ progress was monitored in partnership with them, and a 
range of external accreditation systems was used. Student-centred learning, widening 
participation, empowerment and individual progression were the keynotes of these co- 
ordinators’ philosophy. The predominant factor in students’ success was said to be that 
staff should have training and the relevant attitude towards students - understand their 
needs and be committed to meeting them, both academically and emotionally. 

If there was a stronger than expected theme in the interviews, it was the apparently 
more detailed attention given to staff induction and development. In all seven areas 
professional support was given through appraisal systems with a development officer 
or through training courses - one-off training days and workshops on a range of areas 
including PowerPoint and teaching styles, dyslexia awareness and diagnosis, the new 
basic skills standards, and information and learning technology Guidance for staff on 
materials and methods was provided both through formal sessions and informally. 
Tutors were recommended to use both commercial and in-house materials. In several 
areas less experienced staff, especially newcomers, shadowed or were mentored by an 
established colleague. All of this would appear to support the case for stronger 
professional development for adult literacy tutors. 

6.7 Summary 

• Very few factors associated with differential progress were discovered. 

• For reading, attendance at 51-60 hours of tuition had the strongest association 
with progress. Since regular attendance over several months would have been 
needed for students to attend this number of hours of tuition between pre- and 
post-test, the data suggested that very regular attendance was associated with 
greater progress . 



7 6 Progress in Adult Literacy 



• Since the students on intensive courses attended regularly, this suggests that 
intensive courses could be particularly effective. 

• For reading, the only other factors found to have a positive relationship with 
students’ gains were tutors having assistance in the classroom and all tutors in 
an area having qualified tutor status. 

• In writing, no clear pattern of factors associated with progress emerged. 

• Interviews with co-ordinators in seven areas where students made statistically 
significant progress revealed little that was unexpected, except perhaps a 
tendency towards more detailed attention to staff induction and development. 

• A number of findings suggested a need for greater professionalisation of the adult 
basic skills teaching force. 
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Chapter 7 



Summary and recommendations 



7. 1 Background, aims and method 

• This study began before, but was completed after, the appearance of the Moser 
Report. 

• The Basic Skills Agency commissioned the National Foundation for Educational 
Research to investigate the progress in literacy made by adults in dedicated 
mainstream basic skills provision in England and Wales, and factors associated 
with that progress. 

• The study began in early 1998, and lasted two years. 

• A set of reading tests was compiled from items used in previous studies. The items 
were chosen, and the tests designed, to provide three forms of national data: 
norm-referenced results against a national distribution, criterion-referenced 
results against a set of basic skills standards, and comparisons with school-level 
performance. 

• Two main reading test forms were devised. Half the sample (randomly assigned) 
took each form at pre-test, and the other form at post-test. A complex and 
innovative statistical process was used to equate the test forms. The tests made 
literacy demands which could be considered largely functional; that is, they 
simulated tasks likely to be faced in everyday life. 

• One-sentence writing prompts were also used at pre- and post-test. 

• A total of 2,135 students (representing provision in 71 Colleges of Further 
Education and Local Education Authorities) took the reading pre-test, and 1,224 
also the reading post-test (57 per cent retention). Writing scripts were received 
from 1,724 students at pre-test, and 937 at post-test (54 per cent retention). 
Detailed comparisons showed that those who returned at post-test were fully 
representative of the whole pre-test sample test, in terms of the characteristics on 
which information was available. 

• Most of the students who took part in the study were in ‘normal’, ongoing basic 
skills provision. However, a small number of experimental and highly intensive 
courses were mounted in March 1999, and 206 students who attended those 
courses were studied both in their own right and as part of the full sample. 
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• Background data were gathered on students, and 177 adult literacy tutors 
responded to a questionnaire. 

• Qualitative information was derived from interviews with seven basic skills co- 
ordinators and from fieldworkers’ serendipitous impressions - there was no 
opportunity to carry out systematic observations of teaching. 



7.2 Previous studies 

• There has been only one previous quantitative national study of progress in adult 
literacy in Britain. That study was conducted over 20 years ago, its samples were 
small, and the statistical model used has since been found wanting. If reliable, the 
results suggest that reasonable progress was made in the forms of adult literacy 
provision then in use. 

• None of the national progress studies of adult literacy in the USA provided 
reliable data, while a recent re-analysis of data from a large national one-off 
survey in the USA found no evidence of benefit to literacy attainment of 
participation in adult basic skills provision. 

• It therefore appears that the study reported in this volume is the first in the 
English-speaking world to provide reliable evidence of progress in adult literacy 
based on an adequately representative national sample. 



7.3 Adult literacy teachers and teaching 

• Adult basic skills is a largely part-time and very largely female profession. Tutors 
are predominantly qualified in Arts subjects and mostly teach small groups a few 
hours per week. 

• Adult literacy tutors seemed to keep largely up to date in terms of attending 
courses and reading books relevant to students’ needs, but the writers on literacy 
who were most familiar to them had produced their most influential work some 
time ago. 

• Overall, professional development and management support for tutors appeared 
patchy, and this was consistent with one of the findings of the Moser Report. 

• It seemed that appropriate use was being made of classroom assistants, where 
they were available. No explanation was available of why nearly two fifths of 
tutors had no assistance. 
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• Tutors rated the ethos of their teaching as somewhat more important than attention 
to the practical skills of reading and writing, but the published materials they used, 
and those they devised themselves, did seem to address those skills directly. 

• The overall impression of teaching was one of flexibility and individual initiative. 
This could be seen either as professionalism or as ‘winging it’. 

7.4 The students 

The sample of students in this study: 

• contained significantly more women than men (in the full sample, but among the 
intensive students there were slightly more men than women); 

• represented a range of ages and occupational statuses; 

• contained minorities of people for whom English was an additional language, 
and of ethnicities other than white; 

• were mostly poorly qualified; 

• had mostly attended basic skills provision before (in the full sample; among the 
intensive students just over half were new to adult literacy provision), and a 
minority were attending more than one form of provision in 1998/99; 

• mostly received more than 30 hours of tuition between pre- and post-test; 

• had reading levels quite similar, on average, to a national sample of 9-year-olds; 

• were judged by their tutors to range from very weak to about the national 
average, with a majority already above Entry level. 



7.5 The findings on students’ progress 

• Finer statistical discrimination at the lower end of the reading assessment scale 
was achieved. However, there were still too few reading items which the very 
weakest students could manage. 

• Both the full sample and the intensive students achieved a small but statistically 
significant improvement in reading . 

• In reading, just over half the full sample were already reading at a functional level 
at pre-test, and a few students were already above the national average. 
However, 48 per cent of the full sample had less than functional literacy at pre- 
test, and within these 29 per cent had very low pre-test scores. Even so, these 
findings suggest that an even smaller proportion of adults in greatest need of help 
with literacy are being reached than previously thought. 
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• The ‘turbulence’ seen in the results is an inherent aspect of all cognitive testing, 
but suggests that some students were not securely above the threshold of 
functional level. 

• In the reading tests, students within Entry Levels 1 and 2 of the new Basic Skills 
Standards (29 per cent of the full sample) could not generally manage any of the 
test items. 

• Students within Entry Level 3 of the new Basic Skills Standards could generally 
manage only a few very simple information-retrieval items. 

• Students within Level 1 of the new Basic Skills Standards could generally manage all 
the information-retrieval reading items, but none requiring higher-order skills. 

• Students within Level 2 of the new Basic Skills Standards could generally 
manage all the information-retrieval reading items and most of those requiring 
higher-order skills. 

• Many students seemed to have developed coping strategies for reading non- 
continuous ‘official'-type texts, but could cope less well with continuous texts such as 
newspaper articles. They may not be representative of adults in the lower half of the 
national distribution in this respect. 

• Both the full sample and the intensive students achieved a very small but 
statistically significant improvement in writing in terms of length of script , and 
the full sample also in quality of handwriting . However, these were the only 
statistically significant findings for writing; no significant improvements were 
found in terms of reduction of errors or increase in complexity . 

• Many students in the sample seemed to have distinctly poorer writing than reading 
skills. This phenomenon needs to be investigated further, along with its implications 
for teaching. 

• Several of these findings lead to the conclusion that many students’ literacy 
attainments are uneven, between reading and writing, between types of reading 
material, and from one occasion to another. 

• Some students seemed to attend as much to maintain their skills as to improve 
them. 

• The main finding was that students made average gains in both reading and 
writing that were undramatic but worthwhile . 
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7.6 Factors associated with progress 

• Very few factors associated with differential progress were discovered. 

• For reading, attendance at 51-60 hours of tuition had the strongest association 
with progress. Since regular attendance over several months would have been 
needed for students to attend this number of hours of tuition between pre- and 
post-test, the data suggested that very regular attendance was associated with 
greater progress . 

• Since the students on intensive courses attended regularly, this suggests that 
intensive courses could be particularly effective. 

• For reading, the only other factors found to have a positive relationship with 
students’ gains were tutors having assistance in the classroom and all tutors in 
an area having qualified tutor status. 

• In writing, no clear pattern of factors associated with progress emerged. 

• Interviews with co-ordinators in seven areas where students made statistically 
significant progress revealed little that was unexpected, except perhaps a 
tendency towards more detailed attention to staff induction and development. 

• A number of findings suggested a need for greater professionalisation of the adult 
basic skills teaching force. 



7. 7 Recommendations 

• All new students should be encouraged to attend very regularly , perhaps on an 
intensive course/provision. Intensive provision should perhaps also be offered to 
existing students who want to try getting a quick boost to their skills. 

• Many students will have the further goal of not losing the skills they have gained, 
and these students will need both maintenance of their skills and support for 
their literacy dealings with the world. They will continue to need non-intensive 
provision. 

• In-service professional development for tutors should embed attention to 
subject knowledge of the practicalities of literacy (teaching spelling, phonemic 
awareness and a range of cueing systems, comprehension, etc.) in a broader 
context of up-to-date theory, and aspects of pedagogy specific to literacy should 
receive more emphasis than general principles of teaching. Access to in-service 
provision should be eased for the large majority of the profession who are 
part-time. 
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• Improved professionalisation of adult literacy tutors should be designed to lead to 
more direct teaching of the specifics of literacy. 

• There is a need for a planned series of micro-studies, of which four are suggested 
below. 

• Methods of making classroom assistance more widely available should be 
examined, and the best ways to deploy tutors’ classroom assistants should be 
investigated. 

• Instruments for assessing students’ progress need to be able to detect very small 
steps of progress at the lowest levels. 

• The conditions in which students’ learning is best consolidated and maintained 
need to be understood much better. 

• Reasons for the apparent discrepancy between reading and writing attainment 
should be investigated, along with its implications for teaching. 
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Appendix A 

How the study was carried out - 
full description 

A.l Coverage 

The BSAs statistical report for 1996/97 (Basic Skills Agency, 1999, Table 4, p.3) 
gave the total number of students in basic skills provision in England and Wales in 
the academic year 1996/97 (the academic year two years before that in which this 
study was conducted, but the most recent for which statistics were available) as 
362,449. The same report (Table 6, pp.5-6) gave the numbers of students attending 
FE College or LEA provision in literacy alone or literacy and numeracy combined; 
see Table A. 1 . 



Table A.l: Numbers of students attending literacy and literacy and numeracy provision in 
England and Wales in 1996/97, by sector 





Literacy only 


Literacy & numeracy 


Total 


England 


FE Colleges 


66,232 


47,048 


113,280 


LEAs 


29,553 


11,400 


40,953 


Total 


95,785 


58,448 


154,233 


Wales 


FE Colleges 


3,491 


4,710 


8,201 


LEAs 


1,632 


1,545 


3,177 


Total 


5,123 


6,255 


11,378 


Grand total 


100,908 


64,703 


165,611 


Total FE 






121,481 


Total LEA 






44,130 



However, the figures in Table A.1 were for all forms of provision, and this study was 
intended to cover only dedicated provision in the BSAs sense. The Basic Skills Agency 
report (Table 8, p.7) also gave the percentages of students in dedicated provision as 56.1 
per cent in England and 40.8 per cent in Wales; applying these' to the figures of 154,233 
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and 11.378 respectively gives 86,524 + 4,642 = 91,166 as an estimate of the number 
of students in dedicated adult literacy provision (with or without numeracy) in England 
and Wales in 1996/97. Assuming the same rate of increase in both academic years 
between 1996/97 and 1998/99 as between 1995/96 and 1996/97 (10.2 per cent; 
Basic Skills Agency, 1999, Table 5, p.3), there would have been about 1 10,700 students 
in the relevant forms of provision in 1998/99. The 2,135 students who took the 
reading pre-test in this study would represent nearly 2 per cent of that figure. 

A.2 Instruments for assessing students’ progress 

A. 2.1 Reading 

Very early in the study it became clear that no instrument capable of meeting the 
Agency’s statistical requirements existed. The timescale of the study did not permit 
the creation of a new instrument, however. Therefore a search was made for 
instruments from which suitable items could be drawn. The process by which a set 
of reading tests was devised is described in Appendix B. 

Because the reading tests relied on an AB/BA design it was decided that they would 
be administered on a secure basis at both pre- and post-test by fieldworkers recruited 
and trained by NFER. Eventually, about 50 people were trained and worked as 
reading test administrators. 



A.2.2 Writing 

Two simple, one-sentence prompts were devised, one each for pre- and post-test: 



(Pre-test) 


Please write a bit about what you hope to learn here. 


(Post-test) 


Please write a bit about what you have learnt here. 



These were printed on sheets which had spaces for the student’s name, ID number 
and college or centre. The reading test administrators left these with the students’ 
tutors to be completed as soon as possible after the administrator’s visit and 
returned to NFER. 

A. 3 Background instruments 

A. 3. 1 Student Profile 

This instrument was designed to gather background information on the students, 
and is reproduced in Appendix C. The reading test administrators left the Profiles 
also with the students’ tutors, at pre-test only, to be completed as soon as possible 
after the administrator’s visit and returned to NFER. 
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A.3.2 Hours of tuition form 

One obvious variable that is often found to be related to educational progress is time, 
the amount of relevant teaching students receive between pre- and post-test. A form 
was devised on which test administrators were asked to fill in, before each post-test 
visit, the names and IDs of the students who had been pre-tested at that centre; the 
administrators left the form with the tutor for completion (filling in each post-tested 
student’s number of hours of attendance between pre- and post- test) and return. 
Where students were attending combined literacy and numeracy provision, tutors 
were requested to fill in only the number of hours of literacy provision attended. 

A.3.3 Tutor questionnaire 

A questionnaire (reproduced in Appendix D) was devised for completion by tutors 
covering their background, current basic skills teaching in general, current adult 
literacy teaching in particular, assistance with teaching, management and support, 
teaching and assessment, and professional development. 

This instrument went through many drafts and was ready in May 1999. Where 
post-testing was still in progress, the questionnaire was given to tutors by the test 
administrators; otherwise it was posted. 

A.4 Qualitative data 

The project team was not asked to carry out any systematic observation of adult literacy 
teaching, and there was no opportunity to do so. However, all the test administrators 
observed a good deal informally while on fieldwork, and a few were asked to put their 
impressions in writing. While in no way representative, these accounts provided some 
qualitative background and insights for this report - see, for instance, section A. 7 below. 

Once all the data were analysed for the full sample, they were further analysed at 
provider level, in order to investigate whether students in some areas achieved 
noticeably better progress than in others. Statistically significant progress had been 
made by students in seven areas, and the adult basic skills co-ordinator in each of these 
was contacted and interviewed - but was not told the basis of selection. Again, these 
interviews were not representative but yielded valuable insights - see section 6.5. 

A. 5 Test scoring and data analysis 

In order to ensure consistency, test scoring was carried out centrally, under NFER’s 
direction. The scoring of the reading tasks was done by temporary workers trained 
for the purpose and using explicit and comprehensive marking keys. The analysis of 
the reading data was a very intricate process, and is described in Appendix E. 
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The writing scripts were scored by a small team of markers trained for the purpose 
and using a specially devised marking scheme. An outline of the scheme is given in 
Appendix F. 

Data on students’ background, on hours of tuition, and from the tutor 
questionnaire were first analysed raw, that is, simply in terms of the frequency of 
responses in various subcategories. Then students’ gains in reading and writing 
were related to each of the categories on the Student Profiles and to the number of 
hours of tuition received between pre- and post-test, and gains in reading to selected 
items from the tutor questionnaire. 



A.6 Sampling 

The project statistician advised that a pre-test sample of 1,000 students on each of 
the two main reading test forms (see Appendix B) would be both sufficiently 
representative of the target population and statistically adequate. Statistical 
adequacy meant that a sample of that size would both enable the two test forms 
to be equated (since students were assigned randomly at pre-test to forms A and B) 
and support the attribution of IALS scaled values to the non-IALS items. It was 
therefore decided to aim for a pre-test sample of somewhat over 2,000 and for a 
post-test sample of at least 1,200. The 'somewhat over’ in the pre-test sample was a 
safety margin in case of higher than expected drop-out - the known drop-out rate 
from adult basic skills provision was about 50 per cent (Kambouri and Francis, 
1994). 

In order to locate the sample of students, the Agency allowed the NFER to use 
its database of providers. Every year the Agency conducts a statistical questionnaire 
survey of basic skills provision in England and Wales, and publishes the results 
as a supplement to its annual report. In early 1998, the database contained 
contact details for about 1,100 providers, and the number of students served by 
each in the academic year 1995/96 (the returns for 1996/97 were still being 
processed). From this list all providers other than FE Colleges and LEAs were 
dropped; also all those with student numbers below 100 - this was 
because reaching such small numbers, often in rural areas, would have been 
impractical. This left a pool of about 500 providers. From these about 100 were 
sampled to represent the FE and LEA sectors in Wales and the regions of 
England. Just under 30 declined, or had insufficient or no provision in 1998/99, so 
that the final numbers of providers participating in the study were as shown in 
Table A. 2. 
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Table A.2: Numbers of providers participating, by sector and country 





England 


Wales 


Total 


Colleges of Further Education 


36 


13 


49 


Local Education Authorities 


15 


7 


22 


Total 


51 


20 


71 



The numbers given are for the progress study and the intensive courses combined. 
The 2.2:1 ratio of FE to LEA slightly over-represented the LEA sector - the ratio of 
student numbers shown in Table A.l was 2.75:1. There were nine providers of 
intensive courses; all but two of these also contributed ‘normal' provision to the 
progress study. 

A. 7 Period of testing 

Pre-testing began in October 1998. Because of the need to recruit at least 2,000 
students into the sample, pre-testing continued until May 1999. Meanwhile, post- 
testing began in March 1999 (because some provision ended at Easter), and 
continued until July. 

The length of time taken to gather the test data had several causes. Where 
centres operated drop-in or open learning provision fieldworkers could (in theory) 
use the whole day for testing. More often, however, large centres operated 
three sessions a day (morning, afternoon, evening); small centres, especially in 
rural areas, sometimes provided just one or a very few sessions per week. 
Virtually no provision occurred on Fridays, and less in the afternoons than 
during the morning and evening. The reading tests took on average over 20 
minutes per student to administer. The number of students present when test 
administrators visited was often fewer than eight; some students declined to take the 
test; and it was only sometimes possible to test all the students present. So, even 
working six. hours (all three sessions) a tester could not possibly test more than 
about 20 students per day; and in fact the average per day was nearer ten. Add to 
that the distances to be travelled, and clashes between times when groups of 
students were available, and a good total for a week for one administrator was 30 
students tested. 

A. 8 Amounts of data collected 

The final numbers of students for whom reading, writing, background and hours of 
tuition data were gathered are shown in Table A. 3. 
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Table A.3: Numbers of students on whom data were gathered 





Pre-test 


Post-test 


Retention rate 


Reading 


Progress study 


1925 


1026 


53% 




Intensive courses 


210 


198 


94% 




Total 


2135 


1224 


57% 


Writing 


Progress study 


1537 


786 


.51% 




Intensives 


187 


151 


81% 




Total 


1724 


937 


54% 


Student Profiles 


Progress study 


1709 


n/a 






Intensives 


206 


n/a 






Total 


1915 


n/a 




Hours of tuition 


Progress study 


n/a 


819 






Intensives 


n/a 


163 






Total 


n/a 


982 





In many cases the numbers given in results tables differ slightly from those just 
given; this is due to missing data. 

In addition, 1 77 tutor questionnaires were returned, seven ABS co-ordinators were 
interviewed, and several test administrators wrote up the impressions they had 
gained during fieldwork. 



A.9 Representativeness of post-test samples 

The proportions of intensive students who returned at post-test were so high that 
there could be no doubt that the returners were representative of those who had 
taken the pre-test. However, retention rates were much lower for the full sample. 
Because of this, it was essential to check the representativeness of those who did 
return. This was done by comparing the returners for the reading tests with the full 
pre-test sample for those tests in two ways: according to the distribution of their 
scaled scores, and according to their background characteristics. These compari- 
sons are shown in Tables A.4-5. 



Appendix A - How the study was carried out - full description 9 1 



Table A.4: Comparison of reading test returners with whole pre-test sample by distribution 
of pre-test scores within IALS levels 





Whole pre-test sample 


Returners only 


IALS scaled score band 


IALS levels 


N 


% 


N 


% 


376+ 


5 


0 


O' 


0 


0 


326-375 


4 


34 


2 


14 


1 


276-325 


3 


559 


26 


302 


25 


226-275 


2 


561 


26 


320 • 


26 ’ 


<226 


1 


977 


46 


584 


48 


Total 




2131 


100 


1220 


100 



Table A.5: Comparison of reading test returners with whole pre-test sample by background 
characteristics 





Whole pre-test sample 


Returners only 


N 


% 


N 


% 


Total 




1915 


100 


1134 


100 




Sex 


Men 


819 


45 


498 


45 


Women 


1023 


56 


606 


55 


Total 


1842 


100 


1104 


100 


no information 


73 




30 






Age 


70+ 


17 


1 


16 


2 


65-69 


31 


2 


25 


2 


60-64 


51 


3 


35 


3 


55-59 


74 


4 


42 


4 


50-54 


107 


6 


72 


7 


45-49 


153 


9 


96 


9 


40-44 


183 


11 


97 


10 


35-39 


244 


15 


151 


15 


30-34 


239 


14 


147 


14 
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Whole pre-test sample 


Returners only 


N 


% 


N 


% 


Age 


25-29 


169 


10 


87 


9 


20-24 


156 


9 


76 


8 


under 20 


248 


15 


153 


15 


Total 


1672 


100 


1016 


100 


no information 


243 




118 






Ethnicity 


White 


1556 


84 


963 


87 


Black - Caribbean 


82 


4 


50 


5 


Black - African 


22 


1 


8 


1 


Black - Other 


7 


<1 


1 


<1 


Indian 


28 


2 


8 


1 


Pakistani 


57 


3 


26 


2 


Bangladeshi 


14 


1 


7 


1 


Chinese 


17 


1 


10 


1 


Other 


79 


4 


40 


4 


Total 


1862 


100 


1113 


100 


no information 


53 




21 






First language 


English 


1510 


82 


942 


88 


Other 


334 


18 


132 


12 


Total 


1844 


100 


1074 


100 


no information 


71 




66 






Occupational 

status 


F/t employed 


265 


15 


141 


13 


P/t employed 


215 


12 


126 


12 


F/t self-employed 


16 


1 


7 


1 


P/t self-employed 


15 


1 


9 


1 



O 
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Whole pre-test sample 


Returners only 


N 


% 


N 


% 


Occupational 

status 


Unemployed 


465 


26 


286 


26 


F/t education 


174 


10 


115 


11 


Temp sick/disabled 


52 


3 


30 


3 


Perm sick/disabled 


149 . 


8 


104 


• 10 


Looking after 
home/family 


288 


16 


165 


15 


Other 


163 


9 


106 


10 


Total 


1802 


100 


1089 


100 


no information 


113 




45 






Highest 

qualification 


None 


940 


50 


537 


48 


A Basic Skills cert 


124 


7 


98 


9 


CSE/GCSE 


269 


14 


156 


14 


O-Level 


29 


2 


17 


2 


FE 


475 


25 


298 


26 


A-Level 


16 


1 


8 


1 


Higher education 


38 


2 


13 


1 


Total 


1891 


100 


1125 


100 


no information 


24 




9 






Previous basic 
skills tuition 


None 


639 


36 


352 


33 


In 1998 only 


419 


24 


250 


23 


In 1997 or earlier 


709 


40 


468 


44 


Total 


1767 


100 


1070 


100 


no information 


148 




64 





Comparisons between the full sample and the returners are not shown for 
languages other than English, forms of provision being attended in 1998/99 or 
modes of teaching being experienced in 1998/99 because many of the categories 
were too small to be reliable. 
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On almost all other background variables, and in terms of IALS levels at pre-test, 
the returners were an extremely close match for the full sample, and may safely be 
considered a fully representative subset. 

The only slight exception was first language, where rather more non-native than 
native speakers of English seem to have dropped out between pre- and post-test. In 
fact, this is logical: many non-native speakers are in basic skills provision to bring 
their command of spoken English up to a level satisfactory to them, for their own 
purposes and in order to function in an English-speaking society. All will be fluent 
speakers of their mother tongue, and many will have good or even excellent literacy 
in that language; some will also be literate in English, but without spoken fluency 
When they feel that their command of English has improved enough, some students 
in this category cease to attend. 

Parallel analyses of the representativeness of students from whom writing scripts 
were received at both pre- and post-test yielded very similar results. 
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How the reading tests were devised 



B. 1 Reporting requirements 

The BSAs reporting requirements for this study were that the reading tests should 
indicate how good the students’ progress was in national terms. By ‘in national 
terms’ was meant that the tests should provide: 

• norm-referenced statistics to answer the question ‘How good was the progress 
made by these students in relation to national adult norms for literacy?’ 

• some comparison with the performance of school pupils 

• criterion-referenced statistics to answer the question ‘What proportion of 
the students tested moved up at least one level on a relevant set of literacy 
Standards?’ 

Also, design considerations meant that it would be insufficient to have just one 
test form (any improvement at post-test might be due solely to practice, to having 
seen the test before); two parallel or equatable forms would be necessary 
These would then have to be used in an AB/BA or swop-over design: half the 
students, randomly assigned, would take form A at pre-test and form B at post-test, 
and the other half vice versa (if all students took one form at pre-test and the 
other at post-test, any improvement might be due solely to the second form being 
easier). 

Adult basic skills co-ordinators representing about 30 providers were consulted 
about the project in a series of meetings at the Agency in 1998. These consultations 
were very helpful in preparing the ground for the fieldworkers’ visits; they also 
produced a fifth requirement on the reading tests, namely that each of the parallel 
test forms should have two entry points, one simple and the other very simple, so 
that adult literacy tutors could indicate which level of test each of their students 
should attempt. 

It was concluded very early in the project that no existing instrument could meet 
even one of these requirements, let alone all five. There was also insufficient time to 
develop new tests. It was therefore decided to devise a set of tests by borrowing items 
from existing instruments. 
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B.2 Norm-referenced items 

There was only one existing reading test from which standardised (norm- 
referenced) results for the adult population of Britain had been derived. This was the 
test used in the International Adult Literacy Study (IALS), the British section of which 
was conducted in 1996 (Carey et ah, 1997), using a nationally representative 
sample of adults aged 16 to 65. This test was therefore norm-referenced to a 
relevant population. 

However, because IALS tested adults from the whole range of ability the test 
contained items of a very wide range of difficulty. It also contained items of three 
types, testing Prose, Document and Quantitative Literacy (roughly, tasks based on 
continuous texts, tasks based on non-continuous texts such as timetables, and tasks 
based on text but requiring computation). Neither the more difficult items nor those 
testing quantitative literacy would have been fit for the purposes of this study, but a 
few Prose and Document items were suitable for the target group. 

It was therefore decided to use 25 Prose and Document items (11 tasks) from the 
IALS test. These items already had . values on the scale used in IALS. On the 
assumption that they would behave, statistically, in approximately the same way in 
this study as they had in IALS, it would be possible to use these items to ‘anchor’ the 
other items in this study to the IALS scale, that is, to calculate IALS values for the 
non-IALS items: and then use the scaled values for all the items to calculate 
standardised scores for each student. 

Of the 2 5 Prose and Document IALS items used, 

• 11 had been empirically determined within IALS as being within Level 1 of the 
IALS reporting scale - these were all the Level 1 items from IALS; 

• 10 were IALS Level 2 items; and 

• four were IALS Level 3 items. 

The Level 2 and 3 items were all based on tasks which also had Level 1 items. The 
four Level 3 items were used because there needed to be a few items to stretch the 
more advanced students. 

Though items were borrowed from the IALS test, they still left two problems: there 
were only enough of them to create one test form, not two, each with two entry 
points; and none of the IALS items was simple enough for people with very limited 
literacy. Both problems implied a need to ‘create the lower rungs of the ladder', a set 
of very simple items which would not overface students with very low literacy but 
which would provide detailed statistical discrimination at the lower end of the scale. 
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This was done by borrowing very simple items, some from a school-level test, others 
from criterion-referenced sources. 

B.3 Comparison with school-level performance 

To meet the requirement to provide some comparison with school-level 
performance, three tasks (containing 12 items in all) were borrowed from the tests 
used with 9-year-olds (Population A) in the 1991 Reading Literacy Study (Elley, 
1992) carried out by the International Association for the Evaluation of 
Educational Achievement (IEA). The UK did not take part in that study, but in 1996 
NFER carried out a replication of the age 9 section in England and Wales (Brooks et 
ah , 1996). The three IEA tasks were included in the study to provide a minimal 
indication of how well English and Welsh adults performed compared to 9-year-olds 
in the same countries. One of these tasks was of the IALS Prose type, and another of 
the IALS Document type. The third was based on a short narrative text, and this was 
the only text of this genre used in this study; for statistical purposes it was included 
with the Prose tasks. 



B.4 Criterion-referenced items 

In 1992-93 NFER developed for the BSA a battery of literacy and numeracy tests 
intended to assess adults’ attainment against the BSA’s Communication and 
Numeracy Standards. These tests were used by the BSA as a research instrument; 
for instance, some of the literacy tasks were used in a study it commissioned of the 
competence in English of adults from linguistic minorities (Lost Opportunities , Carr- 
Hill et al. , 1996). That study also used some newly devised tasks, and some from 
other existing sources, and all the tasks used were validated against the BSA 
Communication Standards. 

For the present study five tasks (16 items) from Lost Opportunities were used, 
including some originally developed by NFER, plus three tasks (12 items) from 
those developed by NFER but not used in Lost Opportunities. Thus there was in 
the tests a set of items which were intended to be criterion-referenced against 
the BSA Standards, of which the majority had been empirically validated against 
them. 

All the Lost Opportunities and NFER items used were intended to address either Entry 
Level or Level 1 of the BSA Standards, Just as the IALS items were to be used to 
derive IALS scaled values for the non-IALS items, so the Lost Opportunities and NEER 
items were to be used to anchor other items to the BSA Communication Standards, 
so that all 70 items would be criterion-referenced. 
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B. 5 Compiling the test forms 

The process of item selection produced a set of 70 items (22 tasks) in all: 25 items 
(11 tasks) from IALS, 12 items (three tasks) from IEA, 28 items (eight tasks) from 
Lost Opportunities and/or the NFER test battery, and five items newly devised to fill 
out IALS tasks where relatively long texts had too few questions. The items were 
assembled into two test forms, each with two opening sections and a common main 
section. IALS items, being more difficult than the rest, were used only in the 
common sections, together with new items based on the same texts. IEA, Lost 
Opportunities and NFER items were used in the opening sections. Each opening 
section had 10 items, the common section of Form A 16 items, and the common 
section of Form B 14 items. Thus each student was faced on each test occasion with 
only 26 or 24 items. 

The structure of the tests is shown in Figure B. 1 . 



Figure B.l : Structure of the reading tests 





Form A 


Form B 




Form AL 


Form AH 


Form BL 


Form BH 


Opening 


.1 0 very 


10 simple 


10 very 


10 simple 


section 


simple items 


items 


simple items 


items 








Main section 


1 6 IALS items 


14 IALS items 



There were 11 Prose tasks (including the one narrative) yielding 37 items; and 11 
Document tasks yielding 33 items. These were approximately evenly distributed 
between the test forms. The tasks made reading (and to an extent, see the next 
section, writing) demands which could be considered largely functional; that is, they 
simulated public or other real-world literacy demands likely to be faced in everyday 
life. (For more detail on the tasks, see section 5.4.) 



B.6 Methods of task administration 

All the reading tasks were administered in a way which was as close as possible 
to the original method, that is, the way in which they had been administered 
in previous studies. In all cases the student was required to read the stimulus 
unaided, but in all cases the tester told the student what the task was, even though 
this was also stated on the stimulus sheet. 
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Differences between tasks concerned the modes of response: 

• There were a few items where students had only to circle the response (for 
example, two dates on a calendar); 

• There were several items where the tester read the questions to the student and 
then wrote the student’s answers down verbatim; 

• The 12 IE A items were all multiple-choice: students ticked one of four boxes to 
indicate the response. Here students had to read not only the stimulus text but 
also the questions and the four choices of answer to each question, but still did 
not have to write the answer; 

• All the previous item types occurred in the opening sections of the tests. For the 
items based on IALS tasks (which all occurred in the main sections of the tests), 
the student was required not only to read the stimulus unaided but also to write 
the response unaided. 

Thus in the opening sections of the four test versions, the response modes varied 
from the very simple (circling) to a mode somewhat simpler than having to write the 
response unaided, namely multiple-choice. In no case in the opening sections did 
the student have to write an answer. Where the answer did have to be written down 
in the opening sections, the tester did this (in addition to reading the question to the 
student). There was least support for the students’ literacy in the main sections of 
the tests. 
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Student Profile 



NFER ADULT LITERACY STUDY 1998-99 






NFER 


Student Profile 


Identifier 




Use 

6-10 


Student's Surname (IN CAPS) 






11-30 


Student's Given Name(s) (IN CAPS) 






31-50 


Gender: 


Please circle: Male 


/ Female 


51 




Date of Birth: 


-/ •/ 


52-57 




Time in UK: 


years 


58-59 


Ethnic Group (CIRCLE ONE): 








White 




i 




Black - Caribbean 




2 




Black - African 




3 




Black - other 




4 




Indian 




5 


60 


Pakistani 




6 




Bangladeshi 




7 




Chinese 




8 




Other (please specify) 




9 




Occupational Status (CIRCLE ONE): 








Full-time paid employee 




1 




Part-time paid employee 




2 




Full-time self employed , 




3 




Part-time self employed 




4 




Unemployed 




5 


61 


Full-time education 




6 




Temporarily sick/disabled 




7 




Permanently sick/disabled 




8 




Looking after home/family 




9 




Other (please specify) 




0 





0 

ERIC 
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Language: Is English the only language the student speaks? 

Please circle YES 
NO 

If NO, please state the other language(s) the student speaks: 


NFER 

Use 

62 . 

63-64 

65-66 

67 

68 

69 


Education 

Highest qualification awarded: 

Had the student received any basic skills 
tuition before Autumn term 1998? 

Had the student received any basic skills tuition 
before Autumn term 1 997? 


Please circle 
Please circle 


YES 

NO 

YES 

NO 


Which of the following types of provision is the student receiving in the Autumn 




term 1 998? Please tick all that apply 








Primary literacy/numeracy 




□ 


70 


Adult Training (Training for Work) 




□ 


71 


Training via Youth Credits 




□ 


72 


Literacy/numeracy support 




□ 


73 


Workplace training 




□ 


74 


Family literacy 




□ 


75 


Open learning 




□ 


76 


Drop-in 




□ 


77 


Croup tuition 




□ 


78 


One-to-one provision in College/Centre with tutor 




□ 


79 


One-to-one provision in College/Centre with tutor 








PLUS volunteer support 




□ 


80 


Other (please specify) 




□ 


81 



O 

ERIC 
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Appendix D 

Tutor Questionnaire 



NFER Adult Literacy Progress Study 1998-99 



ID: 



TUTOR QUESTIONNAIRE 
CONFIDENTIAL 



Your name: 

LEA/FE College: Centre: 

Thank you very much for helping us with our research. 

This questionnaire seeks information about: 

1 . You 

2. Your current basic skills teaching in general 

3. Your current adult literacy teaching in particular 

4. Assistance with teaching 

5. Management and support 

6. Planning 

7. Teaching and assessment 

8. Professional development 

Please return to: Greg Brooks 

National Foundation for Educational Research 
The Mere, Upton Park 
Slough SL1 2DQ 

01753 574123 

by 30 June 1999 at the latest 



BSL 



© National Foundation for Educational Research 1999 
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About you 

1.1 Please circle one Male ./ Female 

1 2 

1 .2 What kinds of formal education have you completed? 

Please tick a box for each level of formal education you have completed: 



Master's degree or Doctorate □ 

Bachelor's degree which includes Education CH 

Bachelor's degree □ 

City & Guilds Certificate 9282 □ 

City & Guilds Certificate 9283 D 

City & Guilds Certificate 9284 □ 

City & Guilds Certificate 9285 □ 

A Post experience diploma in adult literacy/basic skills □ 

A Post experience diploma in any other subject D 

PGCE □ 

Teacher's Certificate □ 



1 .3 If you tick any of the first three options, please state the main subject(s) studied 

here: 

Master's degree or Doctorate 

Bachelor's degrees which includes Education 

Bachelor's degree 

1.4 By the end of this academic year, how many years of teaching experience will you 
have had? Please round to the nearest whole number 

Total In adult literacy 

No. of years full-time (i.e. over 35 hours a week) | | | 

No. of years part-time | 
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About your current basic skills teaching in general 

2. 1 Do you now teach basic skills full-time (i.e. over 35 hours a week) or part-time? 

( Please tick one) 
Full-time D 

Part-time D 



2.2 



Whether you teach full-time or part-time, please indicate the total number of hours 
per week that you teach basic skills (that is, contact hours with students, including 
resource-based learning): 



(Round to the nearest whole number) 



contact hours per week 



2.3 Within that total, how many hours do you teach adult literacy? 
(Round to the nearest whole number) 



contact hours per week 



2.4 What is the average size of adult literacy group (that is, number of students present) 
that you teach? (If you teach in drop-in' provision, please give the average number 
of students present per session.) 

I students 



About your current adult literacy teaching in particular 

Please answer questions 3.1 to 4.5 with reference to the adult literacy provision 
on which you teach and which was visited by an NFER fieldworker. 



3.1 

3.2 



Flow many hours per week do you teach this group? 
Flow many students regularly attend this group? 



hours per week 
students 



3.3 Please indicate how many of those students are at each of the following levels of 
the Basic Skills Agency's Communication Standards (please ensure that the total 
equals the number in the answer to question 3.2): 

Level 3 \^_ 

Level 2 | 

Level 1 | 

Entry level | 

below Entry level | 
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3.4 We'd like to know which of the following goals each of these students is working 
towards. Using a register or other list of the students in this group, please fill in one 
line for each student and tick as many columns as apply for each student. (For 
example, if the first student on your list is working only towards self-determined 
goals, tick only that column in the first line): 


Student no. 


Portfolio-based 

accreditation 


Test/exams-based 

accreditation 


Self-determined 

goals 


None of 
these 


1 










2 










3 










4 










5 










6 










7 










8 










9 










10 











3.5 Do any of the students in this group, to your knowledge, have other adult literacy 



teaching during the course of the week? 


YES □ 


If NO, please go to question 3.8. 
If YES, please go to question 3.6. 


NO □ •. 


How many of the students in this group have other adult literacy teaching during the 
course of the week? 




students 


How is record-keeping co-ordinated between you, the students and the students' 
other tutors? 


(Please tick all that apply) 


Records are kept centrally 


□ 


Each student keeps own records 


□ 


You keep records 


□ 


The other tutors keep records 


□ 
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3.8 How often do your students have access to computers and other IT equipment? 

(Please tick one) 



Every session 


□ . 


Some sessions 


□ 2 


Never 


□ 3 



3.9 Do you ever use any textbook(s) or commercially-produced materials in teaching 
adult literacy to this group? 

YES □ 

NO □ 

If NO, please go to question 3.11. 

If YES, please list those you use most frequently: 



3.10 Approximately what percentage of your literacy teaching time with this group is 
based on the materials indicated in the previous question? 

(Please tick one box) 

0-25% □ 26-50% □ 51-75% □ 76-100% □ 

1 2 3 4 

3.1 1 Do you ever use any materials you devise yourself in teaching adult literacy to this 
group? 

YES □ 

NO □ 

If NO, please go to question 4. 1 . 

If YES, please list those you devise most frequently: 




■- y i a 
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3.12 Approximately what percentage of your literacy teaching time with this group is 
based on materials you devise yourself? 

(Please tick one box) 

0-25% □ 26-50% □ 51-75% □ 76-100% □ 

1 2 3 4 

Assistance with teaching 

4.1 Do you regularly have volunteers and/or paid assistants working with you in this 
group? 

YES □ 

NO □ 

If NO, please go to question 5.1 . 

If YES,please go to question 4.2 

4.2 On average, in each session, how many volunteers or paid assistants do you have 
working with you in this group? 

1 volunteers 
1 paid assistants 

4.3 Which aspects of the work do they help you with? 

(Please tick all that apply in both columns) 





volunteers 


paid assistants 


Planning 






Support for individual students 






Support for whole group 






Support for more basic students 






Other (please specify) 







4.4 How do you as the tutor organise this? 
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4.5 On average over a term, how often do volunteers and/or paid assistants attend 
classes with this group? (Please tick one box) 



Almost every session D i 

About % of the sessions □ 2 

About V 2 the sessions D 3 

About '4 of the sessions D 4 

Less than '/ 4 of the sessions D 5 



From here to the end of the questionnaire please answer in general terms (and 
not with specific reference to your current group). 



Management and support 

5. 1 What form does support from your basic skills line manager(s) take? 

(Please tick all that apply) 



No support provided D 

Drop-in provision for tutors D 

Visits to your class D 

Enabling you to observe other tutors D 

Out of hours support D 

Other (please specify) . □ 



5.2 About how often do you have formal meetings with other tutors to discuss and/or 
plan the curriculum/teaching approaches? (Please tick one) 



Never • CH 1 

once a term D 2 

twice a term D 3 

more than twice a term but not weekly D 4 

every week D 5 



5.3 About how often do you have informal meetings with other tutors to discuss and/or 
plan the curriculum/teaching approaches? (Please tick one) 



Never □ 1 

once a term D 2 

twice a term D 3 

more than twice a term but not weekly D 4 

every week D 5 
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Planning 








6.1 


In planning your adult literacy teaching, do you follow a defined curriculum? 








YES □ 
NO □ 








If NO, please go to question 6.3. 
If YES, please go to question 6.2 








6.2 


Is the curriculum you follow 


(Please tick all that apply) 








- devised by you? 


□ 








- based on an accreditation framework? 


□ 








- devised by your Programme Manager? 


□ 








- other? (Please specify) 


□ 






6.3 


In planning your adult literacy teaching, do you follow a scheme of work? 








YES □ 
NO □ 








If NO, please go to question 6.5. 
If YES, please go to question 6.4 








6.4 


Is the scheme of work you follow 


(Please tick all that apply) 








- devised by you? 


□ 








- based on an accreditation framework? 


□ 








- devised by your Programme Manager? 


□ 








- other? (Please specify) 


□ 






6.5 


When planning adult literacy provision, how much do you rely on . . . 










Please circle one number in each row 






not 


a 


a great 






at all 


little 


deal 




(a) your own previously prepared classes? 1 


2 


3 




(b) students' individual learning plans? 


1 


2 


3 




(c) other tutors or ABS specialists in your LEA/College? 1 


2 


3 




(d) students' edition of textbooks? 


1 


2 


3 




(e) other textbooks or resource books? 


1 


2 


3 




(f) tutors' guides or tutor edition of textbook? 1 


2 


3 




(g) external assessments? 


1 


2 


3 




(h) feedback from your observations and assessments 








of students 1 ' work? 


1 


2 


3 



erJc 18$ 
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Teaching and assessment 

7.1 In your adult literacy teaching, how often do you 


include the following 


in your 


scheme of work? 

Please circle one number in each row 


never 


some 


most 


every 


or almost lessons 


lessons 


lesson 


never 

(a) reading aloud 1 


2 


3 


4 


(b) work on phonics, including phonemic 

awareness 1. 


2 


3 


4 


(c) work on spelling 1 


2 


3 


4 


(d) work on punctuation 1 


2 


3 


4 


(e) work on handwriting skills 1 


2 


3 


4 


(f) cloze exercises 1 


2 


3 


4 


(g) matching exercises 1 


2 


3 


4 


(h) work to develop a sight vocabulary 1 


2 


3 


4 


(i) work to develop reading strategies 1 


2 


3 


4 


(j) work to develop spelling strategies 1 


2 


3 


4 


(k) work on grammar/syntax 1 


2 


3 


4 


(1) work on whole language/language 

experience 1 


2 


3 


4 


(m) reading a passage and answering 

questions on it 1 


2 


3 


4 


(n) explaining the reasoning behind an answer 1 


2 


3 


4 


(o) students writing for their own purposes 1 


2 


3 


4 


7.2 For approximately what proportion of the time in your adult literacy classes do you 


teach your students? 

Please circle one number in each row 




never about '/> 


about '/* 


about V 4 


all the 


time 


time 


time 


time 


(a) individually? 1 2 


3 


4 


5 


(b) as a class? 1 2 


3 


4 


5 


(c) in mixed-ability sets or 

small groups? 1 2 


3 


4 


5 


(d) in sets or small groups 

selected by ability? 1 2 


3 


4 


5 
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7.3 This is a double question about assessing the progress of your adult literacy 


students. For each of the forms of evidence listed below please 


(1) circle one of the first three numbers to indicate how much weight you give it in 
planning your students' progress; 


(2) circle one of the second three numbers to indicate how much weight you give it 
in assessing your students' progression in learning: 


Please circle one 


Please circle one 


number in each row 


number in each row 


Planning 


Assessing 


none some a great 


none .. some a great 


deal 

(a) Student self-assessment 


deal 


negotiated with you 12 3 


1 2 3 


(b) Students' portfolios 1 2 3 


1 2 3 


(c) Continuous assessment 1 2 3 


1 2 3 


(d) How well students do on 




written assignments 1 2 3 


1 2 3 


(e) How well students do in oral 




work 1 2 3 


1 2 3 


(f) Listening to students read 




aloud 1 2 3 


1 2 3 


(g) Tutor-made tests 1 2 3 


1 2 3 


(h) Published tests 1 2 3 


12 3 
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7.4 To make progress in literacy, how important do you think it is for students to: 

Please circle one number in each row 







not somewhat 


very 


(a) 


remember spellings? 


important important 
1 2 


important 

3 


(by 


write in an organised way? 


1 2 


3 


(c) 


understand concepts about print? 


1 2 


3 


(d) 


understand how literacy is used in the 
real world? 


1 2 


3 


(e) 


have ownership of their own learning? 


1 2 


3 


(0 


work on topics which are relevant to them? 


1 2 


3 


(8) 


regularly achieve short term goals? 


1 2 


3 


(h) 


be given support, evaluation and tutorials? 


1 2 


3 


(i) 


carry out home study? 


1 2 


3 


0) 


read regularly? 


1 2 


3 


(k) 


develop phonological awareness? 


1 2 


3 


(1) 


develop self-confidence? 


1 2 


3 


(m) 


find their teaching group safe, socially? 


1 2 


3 



Professional development 

8. 1 Please name the book on the teaching of literacy you read most recently: 



8.2 When did you read it? (month and year) | 1 1 

8.3 Have you attended any professional development (in-service) activities (such as 
courses, conferences, seminars) since September 1996? 

YES □ 

NO □ 

If NO, please go to question 8.5. 

If YES, please go to question 8.4 



O 
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8.4 Which areas did the professional development activity/ies cover? 
(Please tick all that apply. ) 



teaching of reading D 

teaching of writing □ 

recent research in literacy □ 

phonological awareness □ 

IT and literacy □ 

Other (please specify) □ 



8.5 For each of the following authors, please tick to show whether or not you are familiar 
with his/her work on literacy: 





Familiar 


Some Knowledge 


Not familiar 


D. Barton 








R. Beard 








N. Bielby 








L. Bradley 








P. Bryant 








H. Dombey 








K. Goodman 








U. Goswami 








M. Hamilton 








S. Lloyd 








M. Meek Spencer 








F. Smith 








M. Snowling 








L. Waterland 









Thank you very much for completing this questionnaire. Please return it to Greg Brooks at 
NFER (see front page) by 30 June 1999 at the latest. 
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How the reading data were analysed 



E.l Norm-referenced data 

Two series of tests were used, labelled A and B. At the pre-test, testers were 
instructed to give out the A and B series alternately. Depending on their tutors' 
assessment of their ability, the more advanced students were given the 'H' test (AH or 
BH), and those whom it was felt might have difficulty with these were given the ‘L’ 
test (AL or BL). At the post-test, respondents were given the corresponding version 
in the other series. Thus for example those who had AH first time round were given 
BH at the post-test, and those who had taken BL were given AL at post-test. 

Within each test, items were labelled as either ‘P’ (Prose) or 'D’ (Document). The 
basic plan was to calibrate the items using a two-parameter logistic model (Lord, 
1980): 



Then the items were equated to the parameters reported for IALS using a linear 
transformation. Numbers of items available for equating in each test were as shown 
in Table E.l. The same IALS equating items appeared in both the ‘H’ and ‘L’ forms of 
each test. 



Table E.l : Number of items available for equating with IALS 



Item Type 


Form A 


Form B 


Total 


Prose 


6 


3 


9 


Document 


7 


9 


16 


Total 


13 


12 


25 



Because of the sampling design, the populations that took the A’ and ‘B’ tests were 
randomly equivalent, and so it was possible to calibrate the A’ and 'B' items jointly. 
This meant that there were 9 IALS Prose items available for equating, and 16 IALS 



P[*i= l OjA.bi) = 1 

1+ exp[ - DaWj-bi)] 



ERIC 
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Document items. These numbers, especially that for Prose items, are rather low 
compared with the recommended number of 20. 

Tables E.2-5 show the item statistics for fitting the ‘P’ and ‘D’ items in the A and ‘B’ 
subtests, respectively. In none of the tables is the fit particularly good. In Table E.2, 
14 of the 20 items do not fit at the 0.05 level overall, 4 out of 6 in the IALS items, 
and 6 out of 14 in the rest. In Table E.3 11 of the 16 items do not fit, including no 
less than 6 of the 7 from IALS, and 5 out of 9 of the rest. In Table E.4, none of the 
IALS items fit, while all of the rest do. Table E.5 is the most successful, with only 4 of 
the 26 items showing lack of fit at the 0.05 level. 

What does this mean? It is important to emphasise that lack of item fit does not in 
itself mean that items are ‘bad items', simply that they do not fit with the rest. 
Moreover, the IALS items were not specifically designed for the type of population 
sampled in this study, being viewed as the lower end of a general continuum, but as 
part of an overall scale, while the remainder were designed specifically for those 
undertaking some kind of adult literacy learning. 

In tables E.2, E.3 and E.5, the biserial coefficients and the slopes are higher for the 
IALS items than for the newer ones. High biserials are generally considered to be an 
important feature of a successful item. The higher slopes could indicate either that 
the IALS items form a more coherent scale than the rest (the ‘new' items), or that 
the new items relate particularly strongly to what is taught in literacy classes, so 
that there is less differentiation among students. In either case, it seems that the 
IALS and the new items behave rather differently. 

Table E.4 differs from the other three. It appears that all the new items fit, and none 
of the IALS ones. However the new ones are essentially too easy for the population, 
with all having facilities of over 0.9. It seems likely that this arises because this is one 
of the most emphasised aspects of literacy courses. This degree of lack of fit may 
mean that the new items simply do not scale with the old ones. 



Equating to IALS international scales 

With the above caveat in mind, we proceed to link the results from this study with 
those of IALS. The problems above may mean that results of such an exercise should 
not be taken as providing completely comparable results between the two, but 
rather a general indication of location of performance overall. However, the same 
tests were employed (in a balanced design) at pre- and post-test, and any scaling is 
simply a monotonic transformation of results. We may still gain an impression of 
whether the programme has given rise to any progress. 





116 Progress in Adult Literacy 



Tests are equated using the non-overlapping common items design for IRT methods 
(Kolen and Brennan, 1995, pp.162-7). Where Scale I and Scale J are two 
2-parameter logistic IRT scales that differ by a linear transformation, and 0\, Oi are 
values on the two scales, then the O-values are related: 

O n — AO n + B 

The item parameters on the two scales are related by 
tt// — &ij 

B~ 

b n = Abu + B 



where A, B are given by 



A = (j(b,) 

<r(b,) 

b„= ix (b,)-Aix(b,) 

(Kolen and Brennan, 1995). 

This is what was done here. Common items on the present study's scale and 
the IALS scale were equated in this way. Parameters on the present scale for 
non-common items, and student abilities, were transformed using the same 
formula. 

This was carried out separately for P and D items. Next, a further linear 
transformation was applied to the P and D scales to calibrate them with the IALS 
NAEP-based metric ranging from 0 to 500. The values of A,B were taken from 
Murray et ah (1998, p.174). 

The two transformations were thus: 

0 K i — 51.67 0 n ) + 269.16 for Prose, and 
Oki — 52.460/ij + 237.50 for Document. 

Corresponding transformations were used to derive item metrics. 
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Table E.2: 'Prose' Items in 'A' Tests 



ITEM 


FACIL 


BISERIAL 


THRESH 


SLOPE 


CHISQ 


Of 


' PVAL 


CBIRD1 


0.73 


0.58 


-0.28 


1.79 


20.70 


5 


0.00 


CBIRD2 


0.69 


0.50 


-0.22 


1.51 


9.40 


6 


0.15 


CBIRD3 


0.54 


0.43 


0.29 


1.29 


20.00 


5 


0.00 


CBIRD4 


0.85 


0.83 


' -0.56 


2.71 


19.80 


3 


0.00 


CBIRD5 


0.93 


0.99 


-0.94 


3.09 


3.40 


3 


0.33 


C0R1Q1S1 


0.77 


0.85 ■ 


-0.92 


2.19 


18.50 


7 


0.01 


UNICEF2 


0.85 


0.94 


-1.24 


2.55 


8.30 


6 


0.21 


B6Q1S1 


0.72 


0.90 


-0.63 


2.73 


12.20 


6 


0.06 


SCEGG2 


0.85 


1.05 


-1.18 


3.40 


8.70 


4 


0.07 


SCEGG3 


0.72 


0.80 


-0.76 


1.94 


40.50 


7 


0.00 


B5Q1S1 


0.69 


0.99 


-0.47 


5.00 


12.00 


2 


0.00 


B5Q2S1 


0.63 


0.82 


-0.36 


2.65 


14.80 


6 


0.02 


B4Q1S1 


0.74 


0.98 


-0.64 


3.99 


4.20 


3 


0.24 


B4Q2S1 


0.35 


0.58 


0.55 


1.79 


13.10 


5 


0.02 


CUSTAR1 


0.91 


0.71 


-2.29 


1.90 


4.10 


3 


0.25 


CUSTAR2 


0.82 


0.73 


-1.73 


1.77 


5.50 


5 


0.36 


CUSTAR3 


0.28 


0.21 


1.81 


0.45 


19.60 


8 


0.01 


CQSAND1 


0.52 


0.58 


-0.48 


1.15 


27.50 


8 


0.00 


CQSAND2 


0.51 


0.87 


-0.43 


2.82 


7.20 


5 


0.21 


CQSAND3 


0.64 


0.91 


-0.82 


2.84 


5.20 


4 


0.26' 



eric 



132 
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Table E.3: 'Document' Items in 'A' Tests 



ITEM 


FACIL 


BISERIAL 


THRESH 


SLOPE 


CHISQ 


DF 


PVAL 


SEEDQ1 


0.89 


0.47 


-1.79 


1.06 


15.20 


4 


0.00 


SEEDQ2 


0.86 


0.49 


-1.37 


1.19 


9.00 


4 


0.06 


SEEDQ3 , 


0.74 


0.42 


-0.78 


1.00 


15.60 


6 


0.02 


SEEDQ4 


0.90 


0.64 


-1.56 


1.35 


4.10 


4 


0.40 


SEEDQ5 


0.86 


0.46 


-1.55 


1.06 


7.10 


5 


0.21 


B2Q111S1 


0.80 


1.01 


-0.86 


6.02 


18.30 


2 


0.00 


B2Q112S1 


0.33 


0.54 


0.65 


1.74 


19.50 


4 


0.00 


B2Q113S1 


0.80 


1.02 


-0.84 


6.94 


3.70 


1 


0.05 


B2Q114S1 


0.78 


1.06 


-0.73 


9.33 


0.70 


0 


1.00 


B2Q115S1 


0.60 


0.68 


-0.33 


1.79 


21.10 


5 


0.00 


B5Q7S1 


0.51 


0.70 


-0.01 


' 2.28 


134.40 


5 


0.00 


B5Q8S1 


0.44 


0.53 


0.24 


1.33 


52.80 


6 


0.00 


CALEN1 


0.94 


0.59 


-2.85 


1.52 


7.70 


2 


0.02 


CALEN2 


0.97 


0.54 


-3.43 


1.41 


2.40 


1 


0.11 


TELE1 


0.85 


0.66 


-1.97 


1.45 


76.80 


4 


0.00 


TELE2 


0.84 


0.63 


-2.00 


1.35 


68.50 


5 


0.00 








2 
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Table E.4: 'Prose' Items in 'B' Tests 



ITEM 


FACIL 


BISERIAL 


THRESH 


SLOPE 


CHISQ 


DF 


PVAL 


LETTR1 


0.96 


1.00 


-1.47 


2.66 


1.30 


2 


0.54 


LETTR2 


0.92 


0.77 


-1.27 


2.09 


1.80 


3 


0.62 


LETTR3 


0.98 


1.38 


-1.54 


4.10 


1.50 


0 


1.00 


LETTR4 


0.97 


1.16 


-1.51 


3.25 


1.10 


0 


1.00 


LETTR5 


0.94 


0.79 


-1.54 


1.93 


0.90 


2 


0.66 


B7Q13S1 


0.68 


0.90 


-0.53 


3.53 


53.60 


3 


0.00 


B7Q14S1 


0.41 


0.70 


0.29 


2.46 


33.00 


4 


0.00 


B7Q15S1 


0.51 


0.75 


-0.01 


2.44 


44.80 


4 


0.00 
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Table E.5: 'Document' Items in 'B' Tests 



ITEM 


FACIL 


BISERIAL 


THRESH 


SLOPE 


CHISQ 


DF 


PVAL 


CALLT1 


0.78 


0.59 


-0.52 


1.81 


4.60 


4 


0.33 


CALLT2 


0.84 


0.74 


-0.66 


2.33 


8.20 


3 


0.04 


CALLT3 


0.89 


0.47 


-1.44 


1.28 


3.90 


4 


0.42 


CALLT4 


0.91 


0.57 


-1.36 


1.56 


3.00 


3 


0.40 


CALLT5 


0.85 


0.70 


-0.70 


2.22 


2.50 


3 


0.48 


ELCT0N1 


0.69 


0.51 


-0.92 


1.03 


28.80 


7 


0.00 


COREQ2S1 


0.83 


0.95 


-1.10 


2.71 


3.40 


5 


0.64 


B4Q121S1 


0.76 


0.80 


-0.89 


2.00 


8.40 


6 


0.21 


B4Q122S1 


0.77 


0.84 


-0.92 


2.15 


3.50 


6 


0.75 


B4Q123S1 


0.73 


0.77 


-0.78 


1.90 


3.90 


6 


0.69 


B4Q124S1 


0.71 


0.80 


-0.67 


2.16 


3.20 


5 


0.68 


B6Q4S1 


0.67 


0.76 


-0.53 


2.23 


7.50 


5 


0.18 


B6Q6S1 


0.58 


0.72 


-0.21 


2.21 


6.70 


4 


0.15 


B7Q1S1 


0.83 


0.93 


-1.07 


2.96 


6.50 


3 


0.09 


B7Q3S1 


0.56 


0.71 


-0.15 


2.43 


10.40 


4 


0.03 


DUTCHS3 


0.75 


0.90 


-0.69 


3.45 


5.00 


2 


0.08 


MEETNG1 


0.65 


0.50 


-1.20 


0.96 


10.40 


6 


0.11 


MEETNG2 


0.90 


0.84 


-2.17 


1.95 


3.20 


3 


0.37 


MEETNG3 


0.96 


0.66 


-3.37 


1.41 


0.20 


2 


0.89 


ADVRTS1 


0.79 


0.39 


-2.36 


0.75 


9.50 


5 


0.09 


ADVRTS2 


0.71 


0.69 


-1.28 


1.41 


0.70 


5 


0.98 


ADVRTS3 


0.73 


0.72 


-1.32 


1.53 


3.80 


5 


0.58 


CISLAND1 


0.70 


0.62 


-1.29 


1.22 


3.30 


5 


0.66 


CISLAND2 


0.80 


0.85 


-1.55 


2.02 


6.00 


4 


0.20 


CISLAND3 


0.61 


0.65 


-0.83 


1.42 


4.90 


5 


0.42 


CISLAND4 


0.67 


0.63 


-1.15 


1.27 


19.00 


6 


0.00 
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E.2 Criterion-referenced data 

Developments while the study was in progress caused a change in the plan for 
criterion-referencing. 

It was originally intended that a small panel of experts would be asked to 
reference all 70 items to the BSA’s Communication Standards; the task would 
be eased by the fact that 28 of the items had been either empirically validated 
against the Standards already (in the Lost Opportunities study) or devised to address 
them. 

However, in 1999, when that criterion-referencing exercise was due to start, the 
government-appointed Moser Working Group on Basic Skills proposed the 
replacement of the BSA’s Standards by a new set. The new Standards were made 
available to this project in draft in the Summer of 1999, and made the criterion- 
referencing task considerably easier. This was because of a design feature of the new 
Standards: some of the boundaries between levels were deliberately aligned with 
boundaries between IALS levels, as shown in Table E.6. 

Table E.6: Alignment of IALS levels and new Basic Skills Standards levels 



New Standards 


IALS 


Level 2 


3 


Level 1 


2 


Entry 3 


}' 


Entry 2 


Entry 1 



‘Entry 3, 2 and 1’ are sub-levels within the Entry Level of the new Standards (the 
old Entry Level had no such subdivisions). For present purposes, the significant 
feature of the two scales is that the new Entry 3 /Level 1 boundary is aligned 
with the IALS Level 1/2 boundary; and the new Level 1/2 boundary is aligned 
with the IALS Level 2/3 boundary. Thus it proved possible to use the scaled 
IALS values for both IALS and non-IALS items to reference all 70 items against 
the IALS levels; and then to use the scaled IALS scores for individual students 
to assign every student to an IALS level (and, by extension, students with scores 
within IALS Level 1 to subdivisions within the new Entry Level). 

The IALS scaled score ranges which corresponded to IALS levels (and the extensions 
to subdivisions within the new Entry Level) were as shown in Table E.7. 
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Table E.7: IALS levels and score ranges, and New Standards levels 



IALS level 


IALS score range 


New Standards 'level 


3 


276-325 


Level 2 


2 


226-275 


Level 1 


1 


176-225 


Entry 3 


126-175 


Entry 2 


0-125 


Entry 1 



With a slight adjustment, it was also possible to relate these levels to other 
qualification frameworks in use in England and Wales, as in Table E.8. 



Table E.8: Approximate comparison of levels in various qualifications frameworks 



IALS 

level 


New 

Standards 


BSA (old) 
Standards 


QCA National 
Framework of 
Qualifications 


Equivalent 

Vocational 

Qualification 


Equivalent level in schools 


3 


Level 2 


Level 2 


Intermediate 


Level 2 NVQ 


GCSE grades A*-C 


2 


Level 1 


Level 1 


Foundation 


Level 1 NVQ 


National Curriculum level 4 


1 


Entry 


Entry 


Entry 




National Curriculum level 2 



Sources: For the IALS and New Standards levels, Table E.7 above; otherwise Moser Report (1999, Table A2, p. 102) 



The adjustment to be kept in mind is that the Entry 3/Level 1 and Level 1/2 
boundaries in the New Standards are somewhat higher than those in the previous 
BSA Standards. As Carey et al. (1997, p. 16) pointed out, IALS Level 1 included 
some tasks rated as within (the lower end of) BSA Level 1 for reading, and not just 
tasks rated as below that level. This is because the IALS levels were not designed to 
correspond directly to BSA levels. Table E.8 therefore has to be read as though there 
were a slight downward shift in requirements between the second and third 
columns, the new and old Basic Skills Standards; this shift may also apply across the 
rest of the Table. 
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How the writing samples were 
analysed 



A scheme for assessing the students' writing was commissioned from and developed by 
Dr Tom Gorman. He directed the only previous quantitative national study of the 
progress of adult literacy students in England and Wales (Gorman, 1981; Gorman and 
Moss, 1979). For the present study he developed the marking scheme and trained the 
small panel of markers; their training included marking a common set of scripts. The 
main points of the marking scheme are reproduced at the end of this Appendix. 

The length of the script in words was noted for two reasons. First, any change in the 
average length of students’ writing would be of interest in itself. Secondly, any change 
in the frequency of particular types of error in students’ writing would also be of 
interest; but many of the scripts were very brief and therefore could not contain more 
than a few errors. Students who wrote more, and who therefore had the opportunity 
to make more errors, would be penalised if errors were simply counted up and reported 
as raw totals. The way to counter this was to allow for the length of the script; this was 
done by dividing the number of errors each student made in a category by the length 
of that student's script in words. 

If a script was returned blank, it was dropped from the analysis. It would not have been 
valid to infer that every such script represented a positive decision by a student not to 
write anything. For one thing, some tutors may have returned blank scripts where 
students had not been present and had therefore not had the opportunity to write 
anything. 

Similarly, where it was clear that a script had been scribed for the student, it was also 
dropped from the analysis. A number of scripts were returned with notes attached 
from the tutor saying that they had been scribed - for entirely valid reasons, for 
example where students had a visual or physical impairment, or had wanted to 
contribute a longer account even though they could not have written more than a few 
words unaided. 

Under Grammar, Style, Spelling and Other Orthographic Conventions the numbers of 
errors were noted. Results were calculated first for each of these four categories 
separately, and then for the total number of errors (by adding together the errors in 
each category). 
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The Handwriting category was an assessment, on a simple three-point scale, of the 
quality of the handwriting, largely in terms of letter formation. Since this was not an 
error category the results were used untransformed, that is, not divided by the 
length of script. 

The NFER Adult Literacy Study 1998-99 - Outline of Marking 
Scheme for Writing 

Code the scripts analytically according to the following scheme. If there is more than 
one error in a word, code both. 



1) ID 

For each student, enter their 5-digit ID. 



2) Length 

Count the number of words in the script and enter in the appropriate column. 

For all further categories, make an entry only if you need to. If a script is accurate/ 
error- free in a particular category, leave the relevant column blank. 



3) Scribed/ blank? 

Enter 1 if it is clear that the script has been scribed for the student. 

Enter 2 if the script is blank. 

If you make either of these entries, do not enter any further codes. 

For the remaining categories, in each relevant column enter the number of errors 
you find. (Continue making entries only if you need to.) 

4) Grammar 

This category covered: cases where a sentence boundary which should be marked 
had not been, or where a sentence boundary which should be not marked had been; 
the number of errors of clause and phrase division; and the number of errors of 
grammatical cohesion, e.g. ‘writing’ for ‘written’; omissions of conjunctions; 
inappropriate use of ‘do’ and of progressive verb forms. 

5) Style 

This category covered: non-deliberate uses of speech forms; inappropriate uses of 
non-standard forms; uses of words or idioms inappropriate to the context. 
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6) Spelling 

This category covered: misspellings (omissions, substitutions, insertions, mistakes 
over the doubling ofletters, etc.) that involve the misplacement of a single letter, or 
the misplacement or omission of two or more letters; also phonetically inconsistent 
grapheme substitutions (in which the meaning and pronunciation cannot be 
derived from the spelling). 



7) Other Orthographic conventions 

This category covered: errors of word division, word-internal punctuation 
(apostrophes, hyphens), non-syntactic punctuation, e.g. unnecessary/omitted 
commas, and capitalisation, e.g. omission at the start of a sentence or in proper 
names and titles. 



8) Handwriting 

In this category three sub-categories were distinguished relating to the control of 
handwriting, as follows: 

Code 3 if the handwriting was readable, with letters that were largely consistent in 
shape and form. 



Code 2 if the handwriting showed lack of systematic distinction between upper and 
lower case letters. 



Code 1 if the handwriting showed lack of skill in letter formation (e.g. difficulties 
relating to letter formation, proportion or orientation). 
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